Inside ‘Master134′: Ad networks’ ‘blind eye’ threatens enterprises

How should security vendors handle ad networks that are repeatedly tied to malvertising campaigns?

Online ad networks linked to the Master134 malvertising campaign and other malicious activity often evade serious fallout and continue to operate unabated.

Rob Wright

Associate Editorial Director – TechTarget – SearchSecurity

The online advertising networks implicated in the “Master134” malvertising campaign have denied any wrongdoings, but experts say their willingness to turn a blind eye to malicious activity on their platforms will likely further jeopardize enterprises.

In total, eight online ad firms — Adsterra, AdKernel, AdventureFeeds, EvoLeads, ExoClick, ExplorAds, Propeller Ads and Yeesshh — were connected to the Master134 campaign, and many of them presented similar explanations about their involvement with the malvertising campaign.

They insisted they didn’t know what was going on and when informed of the malvertising activity, they immediately intervened by suspending the publisher accounts of the malicious actors abusing their platforms. However, none of the ad networks were willing to provide names or account information of the offending clients, citing vague company privacy policies and government regulations that prevented them from doing so.

A cybersecurity vendor executive, who wished to remain anonymous, said it’s likely true that the ad networks were unaware of the Master134 campaign. However, the executive, who has worked extensively on malvertising and ad fraud-related campaigns, said that unawareness is built by design.

“They don’t necessarily know, and they don’t want to know, where the traffic is coming from and where it’s going because their businesses are based on scale,” the executive said. “In order to survive, they have to ignore what’s going on. If they look at how the sausage is made, then they’re going to have issues.”


How the Master134 campaign worked.

The use of various domains, companies, redirection stages and intermediaries make it difficult to pinpoint the source of malicious activity in malvertising schemes. Tamer Hassan, CTO and co-founder of White Ops, a security vendor focused on digital ad fraud, said complexity makes the ecosystem attractive to bad actors like malware authors and botnet operators, as well as ad networks that prefer to look the other way.

There aren’t a lot of ad companies that work directly with malware operators, but there are a lot of ad companies that don’t look at this stuff closely because they don’t want to lose money.

Tamer Hassan

CTO and co-founder, White Ops

“It’s easy to make it look like you’re doing something for security if you’re an ad network,” Hassan said. “There aren’t a lot of ad companies that work directly with malware operators, but there are a lot of ad companies that don’t look at this stuff closely because they don’t want to lose money.”

“Malware Breakdown,” an anonymous security researcher that documented early Master134 activity in 2017, offered a similar view of the situation. The researcher told SearchSecurity that because Propeller Ads’ onclkds.com domain was being used to redirect users to a variety of different malvertising campaigns, they believed “the ad network was being reckless or turning a blind eye to abuse.”

In some cases, Hassan said, smaller ad networks and domains are created expressly for fraud and malvertising purposes. He cited the recent Methbot and 3ve campaigns, which used several fraudulent ad networks that appeared to be legitimate companies in order to conduct business with other networks, publishers and advertisers.

“The ad networks were real, incorporated companies,” he said, “but they were purpose-built for fraud.”

Even AdKernel acknowledges the onion-like ecosystem is full of bad publishers and advertisers.

“In ad tech, the situation is exacerbated because there are many collusion players working together,” said Judy Shapiro, chief strategy advisor for AdKernel, citing bad publishers and advertisers. “Even ad networks don’t want to see impressions go down a lot because they, too, are also paid on a [cost per impression] basis by advertisers.”

There is little indication, however, that these online ad tech companies have changed how they do business.

Lessons learned?

Following the publication of the Master134 report, Check Point researchers observed some changes in activity.

Lotem Finkelsteen, Check Point Research’s threat intelligence analysis team leader and one of the contributors to the Master134 report, said there appeared to be less hijacked traffic going to the exploit kit domains, which suggested the ad networks in the second redirection stage — ExoClick, AdventureFeeds, EvoLeads, ExplorAds and Yeesshh — had either been removed from the campaign by the Master134 threat actors or had voluntarily detached themselves (Yeesshh and ExplorAds closed down the domains used in the campaign sometime in December).

But Adsterra is another story. More than six months after the report was published, Finkelsteen said, there’s been no indication the company has changed its behavior.

Meanwhile, the Master134 campaign changed somewhat in the aftermath of Check Point’s report. The threat actors behind the 134.249.116.78 IP address changed the redirection paths and run traffic through other ad networks, Finkelsteen said.

Aviran Hazum, mobile threat intelligence team leader at Check Point Research, noted on Twitter in September that the campaign had a “new(ish) URL pattern” that moved hijacked traffic through suspicious redirection domains and ad networks like PopCash, a Romanian pop-under ad network that was blocked by Malwarebytes for ties to malicious activity.

AdKernel said it learned a lesson from the Master134 campaign and pledged to do more to remove bad actors from its network. However, a review of several of the domains that bear the “powered by AdKernel” moniker suggests the company hasn’t successfully steered away from suspicious ad networks or publishers.

For example, one ad network customer named AdTriage has a domain called xml.adtriage.com that looks exactly like the self-service portals on the junnify and bikinisgroup sites that were also “powered” by AdKernel. AdTriage, however, doesn’t appear to be a real company — adtriage.com is filled with “Lorem Ipsum” dummy text. On the “About Us” page, the “Meet our team” section has nothing except text that says “Pics Here.” (WhoIs results show the domain was created in 2011, and captures of the sites from that year on Internet Archive’s Wayback Machine reveal the same dummy text.)


AdTriage’s site is filled with dummy text.

Escaping consequences

The recent history of malvertising indicates ad companies that issue denials are quite capable of moving onto the next client campaign, only to issue similar denials and reassurances for future incidents with little to no indication that their security practices have improved.

Check Point’s Master134 report, as well as earlier malvertising research from FireEye and Malwarebytes, doesn’t appear to have had much, if any, effect on the reputations of the five companies. They all appear to be in good standing with the online ad industry and have seemingly avoided any long-term consequences from being associated with malicious activity.

ExoClick and Adsterra, for example, have remained visible through sponsorships and exhibitions at industry events, including The European Summit 2018 and Mobile World Congress 2019.

Online ad companies are often given the benefit of the doubt in malvertising cases such as Master134 for two primary reasons: Ad networks are legitimate companies, not threat groups, and digital ads are easy for threat actors to take advantage of without the help or complicit knowledge of those networks.

But Check Point Research’s team has little doubt about the involvement of the ad networks in Master134; either they turned a blind eye to the obvious signs of malicious activity, Finkelsteen said, or openly embraced them to generate revenue.

Other security vendors have also publicized malvertising campaigns that redirect traffic to known exploit kits. FireEye reported in September that a malvertising campaign used the Fallout exploit kit to spread the GandCrab ransomware to victims primarily in Southeast Asia. According to the report, the malicious ads profiled users’ browsers and operating systems.

“Depending on browser/OS profiles and the location of the user, the malvertisement either delivers the exploit kit or tries to reroute the user to other social engineering campaigns,” the report stated.

It’s difficult to determine exactly how much of the online ad ecosystem has been compromised by malicious or unscrupulous actors, said Adam Kujawa, director of malware intelligence at Malwarebytes.

“Advertising is the reason the internet exists as it does today,” he said. “It’s always going to be very close to the heart of all things that happen on the internet. The reason we see so much adware is because these companies kind of … live in a gray area.”

The gray area can be even murkier on the technical side. Despite being key components in the Master134 campaign, the xml.bkinisgroup.com and xml.junnify.com URLs raise only a few alarms on public URL and file scanning services.

VirusTotal, for example, shows that all 67 malware engines used in the scans rate both domains as “clean,” though the Junnify domain did receive a -28 community score (VirusTotal community scores, which start at zero, represent the number of registered users who vote a file or URL as being safe or unsafe).

Malvertising campaigns like Master134 that use multiple traffic flows and advertising platforms could become increasingly common, according to the Check Point report.

“Due to the often complex nature of malware campaigns, and the lack of advanced technology to vet and prevent malicious adverts from being uploaded onto ad-network bidding platforms,” the researchers wrote, “it is likely we will see more malvertising continue to be a popular way for cybercriminals to gain illegal profits for many years to come.”

Rob Wright asks:

What steps does your organization take to prevent malvertising threats?

Join the Discussion

This was last published in April 2019

‘Master134’ malvertising campaign raises questions for online ad firms

Have you fallen prey to Master134 and what did you do? How should the infosec industry handle persistent malvertising threats?

Malvertising and adware schemes are a growing concern for enterprises. Our deep investigation into one campaign reveals just how complicated threats can be to stop.

Rob Wright

Associate Editorial Director – TechTarget – SearchSecurity

Why were several major online advertising firms selling traffic from compromised WordPress sites to threat actors operating some of the most dangerous exploit kits around?

That was the question at the heart of a 2018 report from Check Point Research detailing the inner workings of an extensive malvertising campaign it calls “Master134,” which implicated several online advertising companies. According to the report, titled “A Malvertising Campaign of Secrets and Lies,” a threat actor or group had compromised more than 10,000 vulnerable WordPress sites through a remote code execution vulnerability that existed on an older version of the content management system.

Malvertising is a common, persistent problem for the information security industry, thanks to the pervasiveness of digital ads on the internet. Threat actors have become adept at exploiting vulnerable technology and lax oversight in the online ad ecosystem, which allows them to use ads as a delivery mechanism for malware. As a result, many security experts recommend using ad blockers to protect endpoints from malvertising threats.

But Master134 was not a typical malvertising campaign.

A tangled web of redirects

Rather than using banner ads as a vector for malware infection, threat actors relied on a different component of the digital advertising ecosystem: web traffic redirection. In addition to serving digital ads, many ad networks buy and sell traffic, which is then redirected and used to generate impressions on publishers’ ads. These traffic purchases are made through what’s known as real-time bidding (RTB) platforms, and they are ostensibly marketed as legitimate or “real” users, though experts say a number of nefarious techniques are used to artificially boost impressions and commit ad fraud. These techniques include the use of bots, traffic hijacking and malicious redirection codes.

Threat actors never cease to look for new techniques to spread their attack campaigns, and do not hesitate to utilize legitimate means to do so.

Check Point Research’s report, ‘A Malvertising Campaign of Secrets and Lies’

According to Check Point Research, part of Check Point Software Technologies, Master134 was an unusually complex operation involving multiple ad networks, RTB platforms and traffic redirection stages. Instead of routing the hijacked WordPress traffic to malicious ads, the threat actors redirected the traffic intended for those sites to a remote server located in Ukraine with the IP address “134.249.116.78,” hence the name Master134. (Check Point said a second, smaller source of traffic to the Master134 server was a PUP that redirected traffic intended for victims’ homepages.)

Then, the Master134 campaign redirected the WordPress traffic to domains owned by a company known as Adsterra, a Cyprus-based online ad network. Acting as a legitimate publisher, Master134 sold the WordPress traffic to Adsterra’s network to other online ad companies, namely ExoClick, EvoLeads, AdventureFeeds and AdKernel.

From there, the redirected WordPress traffic was resold a second time to threat actors operating some of the most well-known malicious sites and campaigns in recent memory, including HookAds, Seamless and Fobos. The traffic was redirected a third and final time to “some of the exploit kit land’s biggest players,” according to Check Point’s report, including the RIG and Magnitude EKs.

The researchers further noted that all of the Master134 traffic ended up in the hands of threat actors and was never purchased by legitimate advertisers. That, according to Check Point, indicated “an extensive collaboration between several malicious parties” and a “manipulation of the entire online advertising supply chain,” rather than a series of coincidences.


The redirection/infection chain of the Master134 campaign.

Why would threat actors and ad networks engage in such a complex scheme? Lotem Finkelsteen, Check Point’s threat intelligence analysis team leader and one of the contributors to the Master134 report, said the malvertising campaign was a mutually beneficial arrangement. The ad companies generate revenue off the hijacked WordPress traffic by reselling it. The Master134 threat actors, knowing the ad companies have little to no incentive to inspect the traffic, use the ad network platforms as a distribution system to match potential victims with different exploit kits and malicious domains.

“In short, it seems threat actors seeking traffic for their campaigns simply buy ad space from Master134 via several ad-networks and, in turn, Master134 indirectly sells traffic/victims to these campaigns via malvertising,” Check Point researchers wrote.

Check Point’s report was also a damning indictment of the online ad industry. “Indeed, threat actors never cease to look for new techniques to spread their attack campaigns, and do not hesitate to utilize legitimate means to do so,” the report stated. “However, when legitimate online advertising companies are found at the heart of a scheme, connecting threat actors and enabling the distribution of malicious content worldwide, we can’t help but wonder — is the online advertising industry responsible for the public’s safety?”

Other security vendors have noted that malvertising and adware schemes are evolving and becoming increasingly concerning for enterprises. Malwarebytes’ “Cybercrime Tactics and Techniques” report for Q3 2018, for example, noted that adware detections increased 15% for businesses while dropping 19% for consumers. In addition, the report noted a rise in new techniques such as adware masquerading as legitimate applications and browser extensions for ad blockers and privacy tools, among other things.

The malvertising Catch-22

The situation has left both online ad networks and security vendors in a never-ending game of whack-a-mole. Ad companies frequently find themselves scrutinized by security vendors such as Check Point in reports on malvertising campaigns. The ad companies typically deny any knowledge or direct involvement in the malicious activity while removing the offending advertisements and publishers from their networks. However, many of those same ad networks inevitably end up in later vendor reports with different threat actors and malware, issuing familiar denials and assurances.

Meanwhile, security vendors are left in a bind: If they ban the ad networks’ servers and domains in their antimalware or network security products, they effectively block all ads coming from repeat offenders, not just the malicious ones, which hurts legitimate publishers as well as the entire digital advertising ecosystem. But if vendors don’t institute such bans, they’re left smacking down each new campaign and issuing sternly worded criticisms to the ad networks.

That familiar cycle was on display with Master134; following Check Point’s publication of the report on July 30, three of the online ad companies — Adsterra, ExoClick and AdKernel — pushed back on the Check Point report and adamantly denied they were involved in the Master134 scheme (EvoLeads and AdventureFeeds did not comment publicly on the Master134 report). The companies claimed they are leading online advertising and traffic generation companies and were not directly involved in any illegitimate or malicious activity.


How the Master134 campaign worked.

Check Point revised the report on August 1 and removed all references to one of the companies, New York-based AdKernel LLC, which had argued the report contained false information. Check Point’s original report incorrectly attributed two key redirection domains — xml.bikinisgroup.com and xml.junnify.com — to the online ad company. As a result, several media outlets, including SearchSecurity, revised or updated their articles on Master134 to clarify or completely remove references to AdKernel.

But questions about the Master134 campaign remained. Who was behind the bikinisgroup and junnify domains? What was AdKernel’s role in the matter? And most importantly: How were threat actors able to coordinate substantial amounts of hijacked WordPress traffic through several different networks and layers of the online ad ecosystem and ensure that it always ended up on a select group of exploit kit sites?

A seven-month investigation into the campaign revealed patterns of suspicious activity and questionable conduct among several ad networks, including AdKernel. SearchSecurity also found information that implicates other online advertising companies, demonstrating how persistent and pervasive malvertising threats are in the internet ecosystem.1

Rob Wright asks:

How should the infosec industry handle persistent malvertising threats?

Join the Discussion

This was last published in April 2019

2019’s top 5 free enterprise network intrusion detection tools

What open source intrusion detection system do you prefer, and why?

Snort is one of the industry’s top network intrusion detection tools, but plenty of other open source alternatives are available. Discover new and old favorites for packet sniffing and more.

Peter Loshin

Site Editor – SearchSecurity

Open source and information security applications go together like peanut butter and jelly.

The transparency provided by open source in infosec applications — what they monitor and how they work — is especially important for packet sniffer and intrusion detection systems (IDSes) that monitor network traffic. It may also help explain the long-running dominance of Snort, the champion of open source enterprise network intrusion detection since 1998.

The transparency enabled by an open source license means anyone can examine the source code to see the detection methods used by packet sniffers to monitor and filter network traffic, from the OS level up to the application layer.

One problem with open source projects is that when leadership changes — or when ownership of a project moves from individuals to corporations — the projects don’t always continue to be fully free to use, or support for the open source version of the project may take a back seat to a commercial version.

For example, consider Snort, first released as an open source project in 1998. Creator Martin Roesch started Sourcefire in 2001 in a move to monetize the popular IDS. But, in the years running up to Cisco’s 2013 purchase of Sourcefire, the concern was that the company might allow the pursuit of profit to undermine development and support of the open source project. For example, Sourcefire sold a fully featured commercial version of Snort, complete with vendor support and immediate updates, a practice that has bedeviled other open source projects, as users often find the commercial entity gives the open source project short shrift to maximize profits.

Cisco has taken a different approach to the project, however. While the networking giant incorporates Snort technology in its Next-Generation Intrusion Prevention System (IPS) and Next-Generation Firewall products, Cisco “embraces the open source model and is committed to the GPL [GNU General Public License].” Cisco releases back to the open source project any feature or fixes to Snort technology incorporated in its commercial products.

What is an IDS and why is it important?

IDSes monitor network traffic and issue alerts when potentially malicious network traffic is detected. An IDS is designed to be a packet sniffer, a system able to monitor all packets sent on the organization’s network, and IDSes use a variety of techniques to identify traffic that may be part of an attack. IDSes identify suspicious network traffic using the following detection methods:

  • Network traffic signatures identify malicious traffic based on the protocols used, the source of the packets, the destination of the packet or some combination of these and other factors.
  • Blocked lists of known malicious IP addresses enable the IDS to detect packets with an IP address identified as a potential threat.
  • Anomalous network behavior patterns, similar to signatures, use information from threat intelligence feeds or authentication systems to identify network traffic that may be part of an attack.

IDSes can be host- or network-based. In a host-based IDS, software sensors are installed on endpoint hosts in order to monitor all inbound and outbound traffic, while, in a network-based IDS, the functionality is deployed in one or more servers that have connectivity to as many of the organization’s internal networks as possible.

The intrusion detection function is an important part of a defense-in-depth strategy for network security that combines active listening, strong authentication and authorization systems, perimeter defenses and integration of security systems.

Snort

Snort, long the leader among enterprise network intrusion detection and intrusion prevention tools, is well-positioned to continue its reign with continued development from the open source community and the ongoing support of its corporate parent, Cisco.

In general terms, Snort offers three fundamental functions:

  1. Snort can be used as a packet sniffer, like tcpdump or Wireshark, by setting the host’s network interface into promiscuous mode in order to monitor all network traffic on the local network interface and then write traffic to the console.
  2. Snort can log packets by writing the desired network traffic to a disk file.
  3. Snort’s most important function is to operate as a full-featured network intrusion prevention system, by applying rules to the network traffic being monitored and issuing alerts when specific types of questionable activity are detected on the network.

Security Onion

Unlike Snort, which is a self-contained application, Security Onion is a complete Linux distribution that packages a toolbox of open source applications — including Snort — that are useful for network monitoring and intrusion detection, as well as other security functions, like log management. In addition to Snort, Security Onion includes other top intrusion detection tools, like Suricata, Zeek IDS and Wazuh.

Infosec professionals can install Security Onion on a desktop to turn it into a network security monitoring workstation or install the Security Onion distribution on endpoint systems and virtual environments to turn them into security sensors for distributed network intrusion monitors.

Wazuh

The Wazuh project offers enterprises a security monitoring application capable of doing threat detection, integrity monitoring, incident response and compliance. While it may be seen as a newcomer, the Wazuh project was forked from the venerable OSSEC project in 2015, and it has replaced OSSEC in many cases — for example, in the Security Onion distribution.

Running as a host-based IDS, Wazuh uses both signatures and anomaly detection to identify network intrusions, as well as software misuse. It also can be used to collect, analyze and correlate network traffic data for use in compliance management and for incident response. Wazuh can be deployed in on-premises networks, as well as in cloud or hybrid computing environments.

Suricata

First released in beta in 2009, Suricata has a respectable history as a Snort alternative. The platform shares architectural similarities with Snort. For example, it relies on signatures like Snort, and in many cases, it can even use the VRT Snort rules that Snort itself uses.

Like Snort, Suricata features IDS and IPS functionality, as well as support for monitoring high volumes of network traffic, automatic protocol detection, a scripting language and support for industry standard output formats. In addition, Suricata provides an engine for enterprise network security monitoring ecosystems.

Zeek IDS

The name may be unfamiliar, but the Zeek network security monitor is another mature open source IDS. The network analysis framework formerly known as Bro was renamed Zeek in 2018 to avoid negative associations with the old name, but the project is still as influential as ever.1

Peter Loshin asks:

What open source intrusion detection system do you prefer, and why?

Join the Discussion

More than a simple IDS/IPS, Zeek is a network analysis framework. While the primary focus is on network security monitoring, Zeek also offers more general network traffic analysis functionality.

Specifically, Zeek incorporates many protocol analyzers and is capable of tracking application layer state, which makes it ideal for flagging malicious or other harmful network traffic. It also offers a scripting language to enable greater flexibility and more powerful security.

This was last published in April 2019

How to improve application security testing when it falls short

What kind of testing have you done to improve your application security?

Application security testing is a critical component of enterprise security. Find out what steps you can take to make sure your testing procedures fit the bill.

Kevin Beaver

Principle Logic, LLC – SearchSecurity

Those of us working in security like to think our efforts are all we need to find vulnerabilities, contain threats and minimize business risks.

I had this mindset early on in my security career. The thought was: Go through the motions; do x, y and z; and that will serve as a solid security foundation. I quickly learned the world doesn’t work that way; action doesn’t necessarily translate into results.

Certain efforts contribute to a security program in positive ways, while others burn through time, money and effort with no return. Yet, as it relates to application security, all is not lost. You can take steps as part of your program that can yield near-immediate payoffs, boost your security efforts and minimize your business risks.

It’s easy to look at application security testing as a science — a binary set of methodologies, tests and tools that can deliver what you need when executed on a periodic basis. The problem is that it’s not true.

Without going into all the details required to run a strong application security program, let’s look at some of the common shortcomings of application security testing and discuss what you should and shouldn’t do as you move forward and improve. The following issues rank among the biggest applications security challenges.

Application security is often lumped into network security. This means application security testing is often part of more general vulnerability and penetration testing. As a result, application security doesn’t get the detailed attention it deserves.

Simply running vulnerability scans with traditional tools isn’t going to get you where you need to be. Organizations need to be running dedicated web vulnerability scanners like WebInspect and Netsparker, proxy tools like Burp Suite and the OWASP Zed Attack Proxy, and web browser plugins. This will enable you to perform the detailed testing necessary to uncover what are often critical web vulnerabilities that would have otherwise been overlooked. Simply running vulnerability scans with traditional tools isn’t going to get you where you need to be.

This issue is easy to resolve by getting all the right people involved and ensuring your testing efforts are properly scoped.

Web applications aside, mobile apps are often overlooked. I’m not sure why mobile app security is sometimes ignored. Mobile apps have been around years and often serve as a core component of a business’s online presence.

Faulty assumptions about mobile app security abound, however, among them the belief that mobile apps offer only a limited attack surface because of their finite functionality, or that the apps themselves are secure because they have been previously vetted by developers or app stores. This perspective is shortsighted, to say the least, and it can come back to haunt developers, security teams and businesses as a whole.

Abandoning web testing because sites and applications are hosted by a third party. This is similar to mobile apps not being property vetted. If you’re not doing the testing, somebody needs to — and it better be the company doing the hosting or management because I can assure you, no one else is — other than the criminal hackers continually trying to find flaws in your environment. The bad guys are probably not going to tell you about what they’ve uncovered until they have you backed into a corner, if ever.

Don’t let bystander apathy drive your application security testing. Be accountable or hold someone else accountable and review the work.

Companies that decline to perform authenticated application testing. It may be difficult to test every possible user role, but you really need to examine all the aspects of your application eventually.

In the application security testing I conduct, I often see multiple user roles with no critical flaws. But when I test one or two more roles, big vulnerabilities like SQL injection surface. An oversight like this — simply because you didn’t have the time or the budget to test everything — will likely prove indefensible. You need to think about how you’re going to respond when the going gets rough with an incident or breach. Better yet, think about how you’re going to prevent an oversight from facilitating application risks in the first place.

If you want to find and eliminate the blind spots in your application security testing, you must do the following:

  • Get the right people involved, including developers and quality assurance
  • Develop standards and policies governing application security.
  • Perform your testing on a periodic and consistent basis, repeatedly over time.
  • Keep management in the know and on your side.

A wise person once said, “Is this as good as you’re going to get, or are you going to get any better? Look at your application security testing program through this lens. Bring in an unbiased outsider if you need to.

You’re probably working in the security field because it has great payoffs — both tangible and intangible. Things change daily, and there’s always something new to discover and learn. Whether you work for an employer or you’re out on your own, if you’re going to get better and see positive, long-term results with application security, you have to be willing to see what you’re doing with a critical eye and assume there’s room for improvement. Odds are, there is.1

Kevin Beaver asks:

What kind of testing have you done to improve your application security?

Join the Discussion

This was last published in April 2019

How infrastructure as code tools improve visibility

Do you think infrastructure as code provides enough visibility? Why or why not?

Visibility into cloud infrastructures and applications is important for data security. Learn how to maintain that visibility while using infrastructure as code tools.

Michael Cobb

CISSP-ISSAP – SearchSecurity

When it comes to understanding how all the elements of a computer network connect and interact, it’s certainly true that a picture — or in this case, a network diagram — is worth a thousand words.

A visual representation of a network makes it a lot easier to understand not only the physical topology of the network, its routers, devices, hubs, firewalls and so on, it can also clarify the logical topology of VPNs, subnets and routing protocols that control how traffic flows through the network.

Maintaining visibility across infrastructures and applications is vital to ensure data and resources are correctly monitored and secured. However, research conducted by Dimensional Research and sponsored by Virtual Instruments showed that most enterprises lack the tools necessary to provide complete visibility for triage or daily management. This is a real concern, as poor infrastructure visibility can lead to a loss of control over the network and can enable attackers to remain hidden.

Infrastructure as code, the management of an IT infrastructure with machine-readable scripts or definition files, is one way to mitigate the security risks associated with human error while enabling the rapid creation of stable and consistent but complex environments. However, it’s vital for you to ensure that the resulting network infrastructures are indeed correctly connected and protected and do not drift from the intended configuration.

Infrastructure as code tools

Infrastructure as code tools, such as Cloudcraft and Lucidchart, can automatically create AWS architecture diagrams showing the live health and status of each component, as well as its current configuration and cost. The fact that the physical and logical topology of the network are created directly from the operational AWS configuration, and not what a network engineer thinks the infrastructure as code scripts have created, means it is a true representation of the network, which can be reviewed and audited.

There are similar tools for engineers using Microsoft Azure, such as Service Map and Cloudockit. Security fundamentals don’t change when resources and data are moved to the cloud, but visibility into the network in which they exist does.

Once a network generated using infrastructure as code tools has been audited and its configuration has been secured, it’s important to monitor it for any configuration changes. Unmanaged configuration changes can occur when engineers or developers make direct changes to network resources or their properties in an out-of-band fix without updating the infrastructure as code template or script. The correct process is to make all the changes by updating the infrastructure as code template to ensure all the current and future environments are configured in exactly the same way.

AWS offers a drift detection feature that can detect out-of-band changes to an entire environment or to a particular resource so it can be brought back into compliance. Amazon Virtual Private Cloud Flow Logs is another feature that can be used to ensure an AWS environment is correctly and securely configured.

This tool captures information about the IP traffic going to and from network interfaces, which can be used for troubleshooting and as a security tool to provide visibility into network traffic to detect anomalous activities such as rejected connection requests or unusual levels of data transfer. Microsoft’s Azure Stack and tools such as AuditWolf provide similar functionality to monitor Azure cloud resources.

Security fundamentals don’t change when resources and data are moved to the cloud, but visibility into the network in which they exist does. Any organization with a limited understanding of how its cloud environment is actually connected and secured, or that has poor levels of monitoring, will leave its data vulnerable to attack.

The tools and controls exist to ensure network engineers and developers can enjoy the benefits of infrastructure as code without compromising security. Like all security controls, though, you need to understand them and use them on a daily basis for them to be effective.1

Michael Cobb asks:

Do you think infrastructure as code provides enough visibility? Why or why not?

Join the Discussion

This was last published in April 2019

Refurbished Enterprise-Class Hard Drives Online-Finding and Buying -An honest guide

Post courtesy of:

The Tekmart Sales Team.

A caveated guide and our opinion:-

The Used Hard Drive Guide

Hard drives are the lifesource of your business, whether on your computers and laptops, or within the network infrastructure of your business through servers, SANS, RAIDs and more.

No matter how well-designed or sturdy a hard-drive may be; all hard drives will eventually fail. Sometimes a drive will show symptoms of an impending fail, allowing users time to back up their data and search for a replacement.

Signs of Hard Drive Failure Include:

  • Sluggish functions
  • Read/write errors
  • Abnormal heat output
  • Whirring, clicking, or other sounds

Other times, hard drives will fail without warning – and that total failure can result in the loss of all data from that particular drive. Data recovery process can be expensive, time-consuming, and result in loss of business, and ultimately may not be successful in recovering hundreds of rands’ worth of digital media, thousands of rands’ worth of customer records, financial records, processes and training documents, or more.

Considerations When Buying a Used Hard Drive

If your business is using legacy equipment, replacement parts may have be EOL by the original manufacturer such as EMC, IBM, Dell, Equallogic, Sun, etc. When this happens, if you do not have any spares on-hand, the used/refurbished market is your best bet for finding a replacement.

If you do not have any contacts in the used market, you may be tempted to turn to eBay. Many reputable used companies sell on eBay, but many parts listed on eBay are sold by liquidation companies who do not have the means to test the equipment they acquire, and possess little knowledge about what they’re listing outside of information presented on the label itself.

The danger from buying hard drives on eBay include:

1. Item may not function
2. Item may be listed incorrectly.
3. Hard drive may not have been wiped.
4. Seller may be overseas, or care little for returning the item or troubleshooting problems
5. Limited stock, seller may not be able to replace equipment
6. Risk further downtime dealing with slow shipping or incorrect/faulty products

How to Buy Used Enterprise Equipment Online

If you’re going to buy used hard drives, or other failed IT equipment, it’s pragmatic to buy from a professional and reputable used IT equipment company. Not only can they test equipment and likely have a quality control program in place, they will also have a DOA return policy in place and offer great customer service.

1. Do a Google (or other search engine) search using the part numbers on your failed hard drive. Hint: Using the manufacturer part number may provide the most accurate search results.
2. Look for professional retail websites that offer secure online purchasing (look for the HTTPS in the url, a shield, or badges from Trustwave, Verisign, etc.)
3. Make sure the product listing has an “Add to Cart” or “Buy It Now” button – not just request a quote!
4. If you’re in a pinch, look for sites that offer same-day or overnight shipping. Be sure to read their shipping and return policies.
5. Look for sites with reviews on the used hard drive you will be buying.
6. Avoid sites that offer “instant quotes” or want you to call for pricing – these can take days waiting for responses and force you to compare prices and options from multiple companies. You can also end up on unwanted mailing lists from data mining.

Conclusion on Buying Used Enterprise Hard Drives Online

Buying one or more previously used hard drives can provide you with a quick and inexpensive way to bring your system back to operational. If you’re buying from a trustworthy, knowledgeable business, buying online can be fast, rewarding, and cost-efficient.

Restarting Navisphere Management Server on EMC CLARiiON CX, CX3, CX4-How-to

Post courtesy of:

The Tekmart Support Team.

It may be necessary to restart the Navisphere management server on an EMC CLARiiON CX, CX3, CX4 if any of the problems below present:

  • A Fatal Event icon (red letter “F” in a circle) is displayed for some physical element of array, but Navisphere CLI reports no faults.
  • Host displays a “U” icon even after rebooting host.
  • Navisphere User Interface (UI) is displaying faults that Navisphere CLI is not showing or are different from what Navisphere CLI is reporting.
  • An unmanaged Storage Processor (SP) still has owned LUNs.
  • Navisphere User Interface (UI) hangs or freezes.
  • Navisphere User Interface (UI) is displaying faults but when the faults option is clicked it shows the array is operating normally.
  • Fault on primary array but all indications shows that the array is operating normally.
  • The Management Servers could not be contacted.
  • Clicking Fault icon returns “array is operating normally” message.
  • CX series array does not recognize the new DAE from Navisphere Manager.
  • Fault after replacing Standby Power Supply

Note: The procedure must be performed on both Storage Processors in order to be effective.

  1. Open a new browser window.
  2. Type in the address bar:  http:// xxx.xxx.xxx.xxx/setupWhere xxx.xxx.xxx.xxx is the IP address of the Storage Processor (SP).
  3. When the screen has loaded, type in the Username and Password used to access Navisphere User Interface (UI).
  4. Once logged in, click the “Restart Management Server” button.
  5. Once the page has loaded, click “Yes”, and then click “Submit.”

Determining EMC Hard Drive Part Numbers and Compatibility-a simple guide

Post courtesy of:

The Tekmart Support Team.

As your EMC CLARiiON, VNX, and AX series grow older, sourcing the exact part number replacements for hard drives can get harder and harder. This guide aims to educate you on how to determine the part number and see compatible part numbers for your system.

Determining EMC Hard Drive Part Numbers

There is a good chance that there are many part numbers listed on a single drive pulled from an EMC array. The generic EMC Model Number does not appear on a drive (e.g. EMC CX-SA07-010 1TB SATA Hard Drive). The disk part number (PN) appears on a label on the front of the disk carrier. This is a nine digit Top Level Assembly (TLA) Part Number like PN 005123456. There are several TLA part numbers that fall under the same EMC model number.

Determining EMC Hard DRive TLA Part Number

Example: Your hard drive has a TLA Part number reading 005048797. Your replacement has a TLA part number 005049070. These are both the same EMC hard drive model number CX-SA07-010 and are hot-swappable.

Finding TLA Part Number in Navisphere

Follow these steps to find TLA part number for a drive in a CLARiiON array:

  1. Open Navisphere by typing in the storage processor’s IP address in a web browser.
  2. Open array with the fault. This is usually indicated by a red “F.”
  3. Open Physical.
  4. Open the Bus and Enclosure with the fault.
  5. Open Disks.
  6. Right-click the disk above or below the disk with the fault and select properties. The TLA part number should be listed at the bottom.

Follow these steps to check and retrieve necessary information for single disk failure:

Check the current status:

1.     Log in to Navisphere manager, right-click CLARiiON name and select “Faults.”

2.     Confirm that the drive x_x_x is the only faulty drive that is  showing as “Removed.”

3.     Expand “LUN Folder” and expand “Unowned LUNs.” Make sure no user LUN is unowned. (It’s normal to see hot spares in the unowned LUNs section.)

Get the TLA Part number of the faulty disk:

1.     Right-click SP A or SP B, select “View Events”, and click “Yes” to continue.

2.     Click “Filter” in the new window, uncheck “Warning” and “Information,” and click “OK.”

3.     Locate Event code “0x7127897c” and Description “Disk(Bus x Enclosure x Disk x) failed,” and double-click to open it.

4.     Record the TLA part number in the description field. It is a 9-digit number starting with “005.”

5.     Refer to following format:

Only one disk failure
No Uknown LUN
Disk Slot: x_x_x
Disk P/N 005xxxxxx

Decoding EMC Model Part Numbers

First two numbers/letters in the EMC model part number indicate the product these drives are for.

CX – CX series
AX – AX series
VX/V2/VS/V3/V4 – VNX series

The next four series of numbers indicate Drive Type and Disk Speed (RPM), or in the case of some Fibre Channel drives Data Rate (GB/s) and Disk Speed (RPM)

2G10 – 2GB/s FC 10K
2G15 – 2GB/s FC 15K
2G72 – 2GB/s FC 7.2K
2S10 – 2.5″ SAS 10K
4G10 – 4Gb/s FC 10k
4G15 – 4GB/s FC 15K
AF04 – 4GB/s FC SSD
AT05 – ATA/SATA 5.4K
AT07 – ATA/SATA 7.2K
FC04 – 4 GB/s FC
LP05 – Low Power FC 5.4K
SA07 – SATA 7.2K
S207 – SATA 7.2K
SS07 – SATA 7.2K
SS15 – SAS 15K
PS15 – VNX SAS 15K
VS07 – VNX SAS 7.2K
VS10 – VNX SAS 10K
VS15 – VNX SAS 15K

The last digits in an EMC part number indicate storage capacity.

73 – 72GB
100 – 100GB
200 – 200GB
250 – 250GB
146 – 146GB
300 – 300GB
320 – 320GB
400 – 400GB
450 – 450GB
600 – 600GB
500 – 500GB
750 – 750GB
900 – 900GB
010 – 1TB
020 – 2Tb
030 – 3TB
040 – 4TB

Changing Bus Speed

If you install a 2 Gb legacy disk in a disk-array enclosure (DAE) on a 4 Gb bus, you cannot use the disk in a RAID group or thin pool until you change the bus speed to 2 Gb. You can change the bus speed with the Backend Bus Speed Reset Wizard, which is available from the Service option on the Navisphere Manager Tools menu. The speed reset operation reboots the storage processors.

EMC Hard Drive DAE Compatibility

Some general rules for EMC hard drive compatibility within the same DAE.

  • You can mix 2GB/s and 4GB/s in a single DAE, but the maximum speed will be 2 GB/s for buses connected to the DAE with both these models of disks.
  • CX-AT, and CX-SA model disks cannot co-exist with other disk models in the same DAE

EMC Hard Drive FLARE-what is this? A brief explanation for basic understanding…

Post courtesy of:

The Tekmart Support Team.

What is FLARE?

In a nutshell:

FLARE (Fibre Logic Array Runtime Environment) OS is a custom Windows Operating System which runs the EMC CLARiiON and VNX Storage Systems. To function properly, disks in an EMC CLARiiON system require that each storage processor run minimum revisions of the FLARE Operating Environment.

The EMC Celerra products use a different version, called DART. EMC Symmetrix / DMX use the Enginuity Code.

FLARE Code version information is as follows:

(Explanation is limited to only to CX, CX3 and CX4 platforms.)

Generation 1: CX200, CX400, CX600

Generation 2: CX300, CX500, CX700 including iSCSI models

Generation 3: CX3-10, CX3-20, CX3-40, CX3-80

Generation 4: CX4-120, CX4-240, CX4-480, CX4-960
(last three digits are the number of drives it can support)

The FLARE Code is a set of 1 number, 2 sets of numbers, a fourth number, and a fifth set of numbers as follows:

1.14.600.5.022 (32 Bit)

2.16.700.5.031 (32 Bit)

2.24.700.5.031 (32 Bit)

3.26.020.5.011 (32 Bit)

4.28.480.5.010 (64 Bit)

First Digit – indicates the number of the machine this code level can be installed on. For the 1st and the 2ndgeneration of machines you should be able to use standard 2nd Generation code levels. CX3 code levels would have a 3 in front of it and so forth.  These numbers will always increase as new Generations of CLARiiON machines are added.

Second Set of Digits  are the release numbers which are very important and provide additional features related to the CLARiiON FLARE Operating Environment. These numbers will always increase, 28 being the latest FLARE Code Version.

Third Set of Digits – The next 3 digits are the model number of the CLARiiON, e.g. CX600, CX700, CX3-20 and CX4-480. These numbers can be anything depending what the model number is.

Fourth Digit – The 5 here is unknown, its coming across from previous FLARE releases. It’s believed that this was some sort of code internally used at Data General indicating its a FLARE release.

Fifth Set of Digits – The last 3 digits are the Patch level of the FLARE Environment. This would be the last known compilation of the code for that FLARE version.

Using Navisphere to Determine FLARE Revision

In Navisphere manager the FLARE OE revision appears on the Software tab of the Storage System Properties dialog box for the system.

  1. Log in to EMC Navisphere – https://ip of the array
  2. Under Storage Management, right-click the storage controller, and select Properties.
  3. Then, select the Software tab to view the FLARE-Operating-Environment revision number.

If you’re using CLI, use the navicli or naviseccli to enter the command “navicli ndu -list -isactive” and get a list of all active software on your array.

If this revision is lower than the minimum FLARE OE revision required for the disk, you must upgrade FLARE OE on the storage system before installing the disk. EMC recommends FLARE OE is upgraded with the CLARiiON Software Assistant in the Navisphere Service Taskbar (NST) or by using the Navisphere Secure Command Line Interface (CLI).

Upgrading FLARE Code on EMC CLARiiON CX4 Array-A step-by-step How to

Post courtesy of:

The Tekmart Support Team.

A caveated introduction

The following is a series of simple documents aimed at providing visual guidance on the process required upgrade the FLARE code on an EMC CLARiiON CX4 array.

  • A version of FLARE 30 is already installed on the array (although the steps for FLARE 29 and below are not significantly different)
  • You have access to EMC’s Unisphere Service Manager (USM) on a Windows platform
  • You have downloaded copies of the FLARE code, Recovery Image and Utility Partition Image.

These documents assume that:

These documents are not intended as a replacement for EMC’s official procedure guides, and should only be used in conjunction with documentation available from Powerlink (http://powerlink.emc.com). You may also refer to http://www.emc.com/cx4support for customized documentation.

The typical warnings apply when upgrading firmware on an array, particularly if it has valuable data on it. Always perform a back-up first.

Process for Upgrading FLARE Code on an EMC CX4 Array

There are 4 basic steps to complete an upgrade:

  • Installing/upgrading Unisphere Service Manager (covered in part 1)
  • Preparing for Installation (covered in part 2)
  • Upgrading FLARE code (covered in part 3)
  • Upgrading the Recovery Image (covered in part 4) and Upgrading the Utility Partition (covered in part 4).  

Part 1: Installing Unisphere Service Manager on an EMC CX4

In the past, there was a software installation wizard that could be launched directly from the CLARiiON, but now the requirement is to use EMC’s Unsiphere Service Manager (USM).

This is a client-side application that can be used to perform basic maintenance tasks, including FLARE code upgrades.

Note: This application was known as Navisphere Service Taskbar (NST) prior to the release of FLARE Release 30.

The first step is to install/upgrade USM on the host that will be used to upgrade the array.

Figure 1 - Launch USM executable file

Figure 1.1 – Launch USM executable file

Figure 2 - Prepare to install

Figure 1.2 – Prepare to install

If an earlier version of USM is detected by the installer it will give you the option to upgrade, it is recommended to click Yes.

Figure 3 - Upgrade option

Figure 1.3 – Upgrade option

Read through the introduction and click Next.

Figure 4 - Introduction

Figure 1.4 – Introduction

Read through the License Agreement and select “I accept the terms of the License Agreement” and click Next to the continue.

Figure 5 - License Agreement

Figure 1.5 – License Agreement

The next step involves the selection of the repository location. This is where USM will store diagnostic data (SP Collects) and items such as FLARE code downloaded from EMC. Use the default options unless there is a particular requirement to store it in a different location.

Figure 6 - Choose Repository Location

Figure 1.6 – Choose Repository Location

Click Next to view the Pre-Installation Summary.

Figure 7 - Pre-Installation Summary

Figure 1.7 – Pre-Installation Summary

After verifying the information, click the Install button. The old version of USM will be uninstalled.

Figure 8 - Uninstalling older version

Figure 1.8 – Uninstalling older version

Figure 9 - Installing Java Runtime Environment

Figure 1.9 – Installing Java Runtime Environment

Figure 10 - Installing Merge Module

Figure 1.10 – Installing Merge Module

Figure 11 - Installing Uninstall Option

Figure 1.11 – Installing Uninstall Option

Figure 12 - Installation Complete

Figure 1.12 – Installation Complete

The Installation process will complete, giving you the option to Launch the USM. USM can also be launched from Start>All Programs>EMC>Unisphere>Unisphere Service Manager>Unisphere Service Manager.

Figure 13 - Launching Unisphere Service Manager

Figure 1.13- Launching Unisphere Service Manager through the Start menu

USM will launch with a default Login screen. Click on the Login button.

Figure 14 - Default Login Screen

Figure 1.14 – Default Login Screen

A dialog box will appear requesting details of the array to connect to. This can be a hostname or IP address. Once entered, click Connect to continue.

Figure 15 - Enter hostname or IP address

Figure 1.15 – Enter hostname or IP address

At this point you’ll then be asked for credentials. You’ll also have the option to select whether you wish to use LDAP authentication, and whether the account you’re using is Local or Global.

Figure 16 - Enter credentials

Figure 1.16 – Enter credentials

This example uses a global, generic account on a test lab array. Once you’ve entered the appropriate credentials, click Login and USM will login to the array.

Part 2: Preparing for Installation

Once login into USM is successful, you’ll be presented with the following screen:

Figure 2.1 - USM - System

Figure 2.1 – USM – System

Click on the Software tab to access System Software, Disk Firmware, and Downloads. Click on System Software.

Figure 2.2 - USM Software Tab

Figure 2.2 – USM Software Tab

There are three choices under System Software: Prepare for Installation, Install Software, and Download and Install Hot Fix. Click on Prepare for Installation.

Figure 2.3 - USM - Software - System Software

Figure 2.3 – USM – Software – System Software

A dialogue screen should pop up with an installation wizard.

Figure 2.4 - Prepare for Installation Wizard

Figure 2.4 – Prepare for Installation Wizard

Select “Verify storage environment only” and click Next, as the software has been installed locally.

Figure 2.5 - Verify storage environment only

Figure 2.5 – Verify storage environment only

Verify once again that the correct array is logged into. Click Next.

Figure 2.6 - Storage System Credentials

Figure 2.6 – Storage System Credentials

Select the software for Installation. Click Browse.

Figure 2.7 - Software Selection

Figure 2.7 – Software Selection

Select the CX4-Bundle that has been downloaded and click Open.

Figure 2.8 - Select CX4 Bundle

Figure 2.8 – Select CX4 Bundle

The software is then unpacked and transferred to the array.

Figure 2.9 - Unpacking Software

Figure 2.9 – Unpacking Software

Figure 2.10 - Transferring Software

Figure 2.10 – Transferring Software

Once the software has been transferred successfully, click Next.

Figure 2.11 - File has been transferred successfully

Figure 2.11 – File has been transferred successfully

IMPORTANT: A manual check is required for a number of conditions (some of which may be obscure) this is very important to read.

Click Next to continue.

Figure 2.12 - Manual Check for Other Conditions

Figure 2.12 – Manual Check for Other Conditions

USM then checks that the hosts attached to the array are capable of failover.

Figure 2.13 - Server Readiness for Software Update

Figure 2.13 – Server Readiness for Software Update

Sometimes, even though the servers are fine, USM won’t be able to accurately gauge the readiness of the servers. Fortunately, there is an Override option, including the standard warning, available to select.

Figure 2.14 - Override HA Status for all servers warning

Figure 2.14 – Override HA Status for all servers warning

Click Next to proceed.

Figure 2.15 - Override HA Status for all servers

Figure 2.15 – Override HA Status for all servers

Next is a diagnostic information step. It is generally recommended to let the USM collect the information again. Select “Collect the diagnostic information again” and click Next. This will take a few minutes to complete.

Figure 2.16 - Collect Diagnostic Information

Figure 2.16 – Collect Diagnostic Information

Figure 2.17 - Diagnostic Information Collection

Figure 2.17 – Diagnostic Information Collection

Once this step is complete, click Next.

Figure 2.18 - Diagnostic Information Successfully Saved

Figure 2.18 – Diagnostic Information Successfully Saved

There are a number of rules that are then checked. If any of the checks fail, there will be an option to fix it. This may take several minutes.

NOTE: If it can’t be fixed by USM, speak with your service provider about a fix before the code upgrade is completed.

Figure 2.19 - Rule Checks

Figure 2.19 – Rule Checks

Once completed, if there are only warnings, you can proceed with the installation assuming the warnings are are acceptable. You may click each warning to see more information about it. Click Next to proceed.

Figure 2.20 - Rule Check Complete

Figure 2.20 – Rule Check Complete

Figure 2.21 - Rule Check Warning Information

Figure 2.21 – Rule Check Warning Information

At this point it is necessary to select the NDU delay. It’s suggested you leave this at the default unless there is a good reason to change it. Click Next to continue.

Figure 2.23 - Non-Disruptive Upgrade Delay

Figure 2.22 – Non-Disruptive Upgrade Delay

Click Finish to complete.

Figure 2.23 - Finished

Figure 2.23 – Finished

Part 3: Installing FLARE Code on an EMC CX-4 Array

Once the installation preparation process is completed, click on Install Software (Step 2).

Figure 3.1 - Install Software

Figure 3.1 – Install Software

As the Prepare for Installation step has already been completed, there is an option to perform an Express Install.

Figure 3.2 - Welcome to the Install Software Wizard

Figure 3.2 – Welcome to the Install Software Wizard

In this example, the Custom Install will be demonstrated to show all the steps that go into upgrading the code. Express installation will be demonstrated further. Click Next to proceed with the Custom Install.

Figure 3.3 - Custom Install

Figure 3.3 – Custom Install

During the previous phase, the software was pre-staged for deployment. Confirm that it is the correct version and click on Next. At this point you could also choose to change the software you’re installing.

Figure 3.4 - Pre-Staged Packages

Figure 3.4 – Pre-Staged Packages

Once again, you’ll need to confirm the HA status of the servers attached to the array.

Figure 3.5 - Server readiness for software update

Figure 3.5 – Server readiness for software update

IMPORTANT: Availability issues with these servers must be addressed before continuing.

Figure 3.6 - Override HA status for all servers warning

Figure 3.6 – Override HA status for all servers warning

Once any availability issues are addressed, select to override the warning and click Next to continue with the installation.

Figure 3.7 - Override HA status for all servers

Figure 3.7 – Override HA status for all servers

USM will check the repository for diagnostic information.

Figure 3.8 - Diagnostic Information Step - Gathering information

Figure 3.8 – Diagnostic Information Step – Gathering information

If not long has passed since the Pre-installation and upgrading the software, it should be safe to use the existing diagnostic information.

Figure 3.9 - Diagnostic Information Step

Figure 3.9 – Diagnostic Information Step

Select “Use the existing diagnostic information” and click Next to continue.

Figure 3.10 - Diagnostic Information Step - Use existing information

Figure 3.10 – Diagnostic Information Step – Use existing information

USM then performs a series of Rules Checks to ensure that everything is in order for the installation to proceed.

Figure 3.11 - Rule Checks

Figure 3.11 – Rule Checks

Note: In this example, a scheduled activity will be interrupted by the NDU. While this isn’t a problem for the lab in the example, one should be mindful of such interruptions in a production scenario. If the results of the Rules Checks are satisfactory and there are no Errors, click Next to proceed to the next step.

Figure 3.12 - Rule Checks Warnings

Figure 3.12 – Rule Checks Warnings

USM then checks that processor utilization on the array is acceptable. The point of this is that, for a period of time while storage processors reboot, all of the array’s workload will be hosted by one SP. If the CPU is already getting belted, it might be wise to re-schedule. This check can be overridden, but it’s not recommended to do this on production arrays.

Figure 3.13 - Processor Utilization Check

Figure 3.13 – Processor Utilization Check

In this example, the lab array is doing sweet FA, so click on Next to continue.

Figure 3.14 - Acceptable Processor Utilization

Figure 3.14 – Acceptable Processor Utilization

Set the NDU delay here.

Note: The default it is set to 360 seconds, and it is strongly recommended that this setting is not changed unless there is good reason to.

Figure 3.15 - Non-Disruptive Upgrade Delay

Figure 3.15 – Non-Disruptive Upgrade Delay

On the next screen you’ll be notified that the ESRS IP Client (also known as CLARalert) will be disabled until the upgrade is complete to prevent false positive alerts to EMC’s triage staff when SPs reboot.

Figure 3.16 - ESRS IP Client Notification

Figure 3.16 – ESRS IP Client Notification

USM is ready to go, so click Next to continue.

Figure 3.17 - Confirmation

Figure 3.17 – Confirmation

This step will take awhile to complete.

Figure 3.18 - Software Maintenance Status

Figure 3.18 – Software Maintenance Status

Click on Show Steps to see the steps.

Figure 3.19 - Software Maintenance Status - Show Steps

Figure 3.19 – Software Maintenance Status – Show Steps

Figure 3.20 - Software Maintenance Status - Further Progress

Figure 3.20 – Software Maintenance Status – Further Progress

At this point it’s rebooting the Secondary SP (SP B). It always does it on SP B first, in case there’s a problem.

Figure 3.21 - Software Maintenance Status - Further Progress

Figure 3.21 – Software Maintenance Status – Further Progress

The following shows completion with all green check marks.

Figure 3.22 - Software Maintenance Status - Complete

Figure 3.22 – Software Maintenance Status – Complete

At this point USM offers to commit the FLARE code on the array. While the array is running new code at this stage, if the software is not committed, it is possible to roll the software back.

Note: While you can service I/O while FLARE is not committed,
you can’t do potentially useful things such as bind LUNs, RAID Groups, and Storage Pools. Unless there’s an obvious problem, it’s recommended that you commit to the package. Before you commit the FLARE code, you should check whether the LCC firmware has been successfully upgraded. On a small array with a few DAEs, this won’t take too long. On a larger array (a CX4-960 with 64 DAEs for example) this might take a little longer.

If there are no storage configuration tasks on the horizon, leave it a day or two and make sure there’s no obvious problems. If it’s been upgraded to Release 30.524 to support 3TB drives on the array, and the resources to shut down the drives the moment the code is committed, then you might not be able to wait that long.

Figure 3.23 - Post-install Tasks

Figure 3.23 – Post-install Tasks

In USM, go to the Diagnostics section. Under the Tools section on the right-hand side of USM, select the “LCC and Power Supply Firmware Update Status” option.

Figure 3.24 - USM - Diagnostics - Diagnostics

Figure 3.24 – USM – Diagnostics – Diagnostics

This screen provides information on the status of LCC Firmware (FRUMON) updates that are kicked off by installing new versions of FLARE. Not every version of FLARE has new LCC firmware, but it’s always a good idea to check.

Figure 3.25 - LCC Firmware (FRUMON) Status

Figure 3.25 – LCC Firmware (FRUMON) Status

Click on “Show details” to see LCC revisions. Depending on the number and type of DAEs, the time it takes to complete this operation will vary greatly, and can be time consuming.

Figure 3.26 - LCC Firmware (FRUMON) Status

Figure 3.26 – LCC Firmware (FRUMON) Status

Once the LCC firmware is complete and everything is working as expected, you can commit the FLARE code. To do this, click on the Run button in the Post-install Tasks screen. A warning about write cache will appear, which should not be problematic when completed during a “quiet” period on the array. This process will take awhile to complete.

Figure 3.27 - Commit Packages - Confirm

Figure 3.27 – Commit Packages – Confirm

Figure 3.28 - Commit Packages - Progress

Figure 3.28 – Commit Packages – Progress

Once it’s complete, there will be a green tick in the Details column and and you may click Next to continue.

Figure 3.29 - Commit Packages - Commit successful

Figure 3.29 – Commit Packages – Commit successful

The Finish screen provides information on the completed activities and confirms completion. If your array is registered for support with EMC or a third-party support provider, you can automatically notify them of the upgrade at this point.

Figure 3.30 - Finish

Figure 3.30 – Finish

Part 4: Installing the Recovery Image and Utility Partition on an EMC CX4

Installing new Recovery Image and Utility Partition is much easier and less time-consuming than upgrading the FLARE.

Installing the Recovery Image on an EMC CX4

To start, click on Install Software. Click Next to continue.

Note: As you have done the Prepare for Installation steps, you won’t have the option to do an Express Install.

Figure 4.1 - Custom Install

Figure 4.1 – Custom Install

This is basically the same process that you followed for installing the CX4 Bundle, the only difference is selecting different packages. Click on Browse to browse for the Recovery Image software.

Figure 4.2 - Software Selection

Figure 4.2 – Software Selection

Figure 4.3 - Select Recovery Image

Figure 4.3 – Select Recovery Image

Once the file has been transferred, click on Next to continue.

Figure 4.4 - File has been transferred

Figure 4.4 – File has been transferred

The installation of the Recovery Image and the Utility Partition doesn’t involve any reboots of the SPs, so you’ll find there aren’t quite as many steps to go through. In fact, just verify the correct version of the Image and click Next to continue.

Figure 4.5 - Express Install Information Verification

Figure 4.5 – Express Install Information Verification

USM loads up the Recovery Image in 12 steps.

Figure 4.6 - Express Install Progress

Figure 4.6 – Express Install Progress

This might take awhile.

Figure 4.7 - Express Install Progress

Figure 4.7 – Express Install Progress

Once done, click Next to finish.

Figure 4.8 - Complete

Figure 4.8 – Complete

At the Finish screen you’ll get confirmation from USM that the software was installed successfully.

Figure 4.9 - Finish

Figure 4.9 – Finish

Installing the Utility Partition

Installing the Utility Partition is roughly the same process as installing the Recovery Image. Select the appropriate version of the software to install.

Figure 4.10 - Select Utility Partition

Figure 4.10 – Select Utility Partition

Verify the correct software and click Next.

Figure 4.11 - Express Install Information Verification

Figure 4.11 – Express Install Information Verification

This step shouldn’t take quite as long.

Figure 4.12 - Complete

Figure 4.12 – Complete

Once it’s done, you’ll get confirmation from USM. Click on Next to finish and complete the process.

Figure 4.13 - Finish

Figure 4.13 – Finish

At this point, if you’re running Unisphere agents on your hosts, you might look at updating them.