1. Prelert Blog

    1. Machine Data is Different (and Why It Matters)

      Explore Anomaly Detection Analytics (Oct 13 2014)

      Machine Data is Different (and Why It Matters)

      Why is machine data different? To answer this question, let’s start by considering different perspectives on what constitutes unstructured data. In the case of a pre-defined set of allowed classifications, the standard approach for benchmarking a machine-learnt classification against a human-generated correct result is to use a confusion matrix . Each of the allowed classifications corresponds to a row and a column in the matrix, and the cells in the matrix record the number of input messages with corresponding human-generated classification and machine-determined classification. The perfect outcome is for all cells of the matrix to contain zeroes except those on ...

      (Read Full Article)

      Comment Mentions:   Prelert

    2. Ready to Talk Anomaly Detection & Advanced Math at Splunk's User Conference

      Explore Anomaly Detection Analytics (Oct 6 2014)

      Ready to Talk Anomaly Detection & Advanced Math at Splunk's User Conference

      The Prelert team, along with partners and customers, will share insights on using machine-based anomaly detection to find value in Big Data in front of over 4,000 IT and business professionals at Splunk’s fifth annual Worldwide Users’ Conference, .conf2014. The event will take place from October 6-9 at the MGM Grand in Las Vegas, Nevada.

      (Read Full Article)

      Comment Mentions:   Prelert   Big Data   Dr. Steve

    3. Anomaly Detection to Reduce the Noise

      Explore Anomaly Detection Analytics (Oct 2 2014)

      Anomaly Detection to Reduce the Noise

      If you have followed some of my other recent blogs, you’ll have noticed that automated anomaly detection is a great technique to find anomalous behaviors in data by effectively contrasting the difference between “normal” and “abnormal. " Most people equate this with contrasting between “good” and “bad,” but that isn’t always necessarily true. What if the data set you’re looking at are “all bad things,” such as Intrusion Detection (IDS) alerts?

      (Read Full Article)

      Comment Mentions:   IT Security

    4. Will You be Replaced by Machine Intelligence?

      Explore Anomaly Detection Analytics (Oct 1 2014)

      Will You be Replaced by Machine Intelligence?

      While humans are definitely needed for the expertise-dependent and creative functions, many aspects of IT operations and performance management could be done more effectively by machine intelligence. Here are just a few examples.

      Deciding What to Monitor
      Most application or service delivery environments have way more metrics, logs and event data than humans can reasonably get their heads around....

      Identifying Normal Behavior
      Even for the 1% of the data we do utilize, it is obvious that setting thresholds and alarm rules is a flawed approach...

      Finding Causal Relationships
      A recent survey by TRAC Research of APM pros turned up the ...

      (Read Full Article)

      Comment Mentions:   Application Performance Management   APM

    5. How to Detect (and Resolve) IT Ops/APM Issues Before Your Users Do

      Explore Anomaly Detection Analytics (Sep 26 2014)

      How to Detect (and Resolve) IT Ops/APM Issues Before Your Users Do

      As originally published by APMdigest. Among the most embarrassing situations for application support teams is first hearing about a critical performance issue from their users. With technology getting increasingly complex and IT environments changing almost overnight, the reality is that even the most experienced support teams are bound to miss a major problem with a critical application or service. One of the contributing factors is their continued reliance on traditional monitoring approaches.

      (Read Full Article)

      Comment Mentions:   APM

    6. Automated Anomaly Detection: A Connector for Amazon CloudWatch

      Explore Anomaly Detection Analytics (Sep 24 2014)

      Automated Anomaly Detection: A Connector for Amazon CloudWatch

      Amazon CloudWatch is a monitoring service for AWS cloud resources and the applications you run on AWS. You can use Amazon CloudWatch to collect and track metrics, collect and monitor log files, and set alarms. At the time of writing, CloudWatch is currently available to all AWS users, with the free version giving basic monitoring metrics (at 5 minute frequency) and generous usage limits. You can also add up to 10 custom metrics and 10 alarms. In this blog I shall explain why it is important to use unsupervised machine learning to effectively manage your AWS environments, and then point ...

      (Read Full Article)

      Comment Mentions:   AWS

    7. Rogue User Detection via Behavioral Analysis

      Explore Anomaly Detection Analytics (Sep 22 2014)

      Rogue User Detection via Behavioral Analysis

      Finding “rogue users” or “rogue systems” using behavioral analysis and automated anomaly detection takes a different approach than the traditional methods of manual data inspection, or the application of rules or signatures to identify specific behavioral violations. A “rogue” user or system, by definition, is someone or something that acts differently from the rest of the population. Therefore, using automated anomaly detection to find behavioral outliers via a comparison of users against each other (peer or behavioral analysis) is a viable approach...

      (Read Full Article)

      Comment Mentions:   IT Security

    8. Prelert Takes Home a Silver Stevie Award

      Explore Anomaly Detection Analytics (Sep 15 2014)

      Prelert Takes Home a Silver Stevie Award

      Last Friday marked the twelfth annual American Business Awards and Prelert was honored with a Silver Stevie Award in the New Product or Service of the Year - Software - Big Data Solution category. The announcement was made at the organization’s first ever New Product & Tech Awards banquet at the Palace Hotel in (where else but the tech mecca) San Francisco...

      (Read Full Article)

      Comment Mentions:   Prelert   Big Data

    9. It's Time to Democratize Data Science!

      Explore Anomaly Detection Analytics (Sep 11 2014)

      It's Time to Democratize Data Science!

      Can we realize our full potential by continuously improving advanced analytics that can only be used by data scientists? Is the right answer found in the 2013 prediction from a leading industry analyst that we need to focus our resources on educating millions of data scientists? There is no way that is sustainable. But there is an answer.  Data science can be packaged for the masses – and that is where our focus should be. Want to know how it's done?

      (Read Full Article)

      Comment Mentions:   Intel   Analytics

    10. Why What You Don't Know May Hurt You, & How Security Analytics Can Help

      Explore Anomaly Detection Analytics (Sep 10 2014)

      Why What You Don't Know May Hurt You, & How Security Analytics Can Help

      Attackers try hard to mask their activities and fly below the radar of your security paradigm – but try as they might, in order to accomplish their goals, their behaviors are going to have to be anomalous at some point in time. An authorized login is going to be attempted from a new IP address. A server is going to run a different process than usual. An unusual pattern of data transmissions will occur to a new external URL. The key to mitigating this threat is to be able to identify these ‘fingerprints’ amidst the billions of records produced by the ...

      (Read Full Article)

      Comment Mentions:   IT Security   Analytics

    11. How Security Analytics Help Identify and Manage Breaches

      Explore Anomaly Detection Analytics (Sep 3 2014)

      How Security Analytics Help Identify and Manage Breaches

      Statistical techniques are the only approach that can identify unknown attacks, and even when applied properly will still require a certain amount of human intervention. Security teams can definitely react a lot faster if they are immediately aware of previously unknown threats, so staying ahead of the bad guys really comes down to two things: the speed of a real-time analysis solution and the reaction time of the security team. In the end, this requires that both the right technology and organizational processes are in place...

      (Read Full Article)

      Comment Mentions:   IBM   Dr. Steve   IT Security

    12. Occupy Your Data. Anomaly Detection Stops the Top 1% from Ruling IT.

      Explore Anomaly Detection Analytics (Aug 27 2014)

      Occupy Your Data. Anomaly Detection Stops the Top 1% from Ruling IT.

      How much of your data do you actually pay attention to?  Would you be surprised to realize it is probably far less than 1%?  How about 1% of 1%? This is the case in the vast majority of IT operations, performance management and security shops of any size anywhere in the world. But a typical web app involves hundreds if not thousands of components including software, networks, middleware, app servers, databases, etc. Now consider what happens when something breaks. Most of the time, one of the KPI you've selected triggers an alert or one of the dashboards you ...

      (Read Full Article)

      Comment Mentions:   Application Performance Management

    13. Data Exfiltration Detection via Behavioral Analysis

      Explore Anomaly Detection Analytics (Aug 21 2014)

      Data Exfiltration Detection via Behavioral Analysis

      There are many possible ways that one can detect “data exfiltration” (data theft), but in many cases, this involves either manual raw data inspection or the application of rules or signatures for specific behavioral violations. An alternative approach is to detect data exfiltration using automated behavioral anomaly detection using data that you’re probably already collecting and storing, and without the use of a DLP-specific security tool.

      (Read Full Article)

      Comment Mentions:   IT Security

    14. The Secret to Fixing Problems Before Users Find Them (part 2)

      Explore Anomaly Detection Analytics (Aug 19 2014)

      The Secret to Fixing Problems Before Users Find Them (part 2)

      In part 1 of this post , we talked about the failed paradigm of using thresholds and rules or 'eyeballs on timecharts' to monitor a critical app or service. In part 2 of this post we'll cover Anomaly Detection products that can leverage data you've already aggregated in stores like Splunk, Elasticsearch or NoSQL databases...

      (Read Full Article)

      Comment

    15. Choosing bucketSpan Wisely

      Explore Anomaly Detection Analytics (Aug 14 2014)

      Choosing bucketSpan Wisely

      In a previous blog post about optimizing the performance of the Engine API, I mentioned that choosing the proper bucketSpan results in not only a possible performance improvement, but I also alluded to bucketSpan affecting the timeliness and quality of your results. In effect, there is a 3-way balance between performance, timeliness of the results, and quality of the results that I’d like to dig into further here...

      (Read Full Article)

      Comment

    16. Static code analysis for C++

      Explore Anomaly Detection Analytics (Aug 12 2014)

      Static code analysis for C++

      Static code analysis has long been touted as a must have for high quality software. Unfortunately, my experience with it in previous jobs didn't live up to the hype. Within the last few years the majority of compilers have added a built-in static code analysis capability, so I thought it would be interesting to see how good they are...

      (Read Full Article)

      Comment

    17. Machine Learning, Anomaly Detection, and the Smart City

      Explore Anomaly Detection Analytics (Aug 7 2014)

      Machine Learning, Anomaly Detection, and the Smart City

      Burdened by heavy traffic, a major metropolitan city wanted to find a solution to help them improve travel times and congestion, and to minimize the effects of incidents and collisions on traffic. Since the city tracks accidents, events, construction, and other road problems, this kind of analysis can be done with automated anomaly detection. By analyzing the traffic and incident data, anomaly detection software can prioritize incidents so that problems with the most impact on traffic are addressed first...

      (Read Full Article)

      Comment Mentions:   Prelert

    18. The Secret to Fixing Problems Before Users Find Them (Part 1)

      Explore Anomaly Detection Analytics (Aug 4 2014)

      The Secret to Fixing Problems Before Users Find Them (Part 1)

      According to a TRAC Research survey on IT performance management challenges the top two issues were 1) 'Problems reported by end-users before IT finds them', and 2) 'too much time spent troubleshooting.' Despite crazy advances in every other field of IT technology, this problem really hasn't changed much in the last 20 years! The good news is we can show you how to change things through the following 3 incremental steps...

      (Read Full Article)

      Comment Mentions:   Google   Application Performance Management   Troubleshooting

  1. Recent Articles for IT Ops & APM

    1. Differences Between Mobile and Server Performance Monitoring

      Explore APMdigest (Oct 15 2014)

      According to eMarketer, as of 2014 Americans consume more media using mobile devices than laptops and desktops combined. This shift in consumer behavior is also occurring within corporations, as employees increasingly rely on mobile devices for their work. With such a surge in mobile usage there is a growing need for corporations to ensure that their mobile experience is high quality and not broken. The following are 4 key differences that companies monitoring their server (and website) performance should consider when selecting a mobile app performance monitoring solution.

      (Read Full Article)

      Comment Mentions:   Application Performance Monitoring

    2. Pursuing DevOps ROI

      Explore APMdigest (Oct 7 2014)

      As big data and the Internet of Things emerge, DevOps becomes even more important. In a software-defined environment, the precision of the software and services controlling networks, sensors and devices is critical as everything becomes inter-connected. The more available and reliable software is the greater the insights from operational intelligence. The greater the insights, the better the decision outcomes and ROI benefits to the organization. This makes pursuing DevOps ROI compelling to all organizations.

      (Read Full Article)

      Comment Mentions:   SaaS

    3. IT analytics tools bring big data to work in the data center

      Explore Data Center information, news and tips (Oct 6 2014)

      As IT environments grow in complexity, scale and heterogeneity, they generate more and more metadata, forcing many organizations to take a purely reactive approach to IT management. For example, with thousands of virtual machines to monitor, it can be difficult to even determine what you need to know when you put a new application into an environment in terms of security, performance or compliance...

      (Read Full Article)

      Comment Mentions:   Google   Gartner   Enterprise Management Associates

    4. Cut big data down to size

      Explore Betanews (Oct 6 2014)

      Identifying complex anomalies that show up the presence of a persistent threat or pinpoint the cause of network and IT performance problems is the sort of thing big data analysis was made for. The problem though is that big data is often just too big. This means it's usually necessary to transfer the information before doing any work on it which makes real-time analysis impossible and means valuable insights aren't available when they're most needed...

      (Read Full Article)

      Comment

  2. Recent Articles for Security Analytics

    1. Use ELK To Visualise Security Data

      Explore elasticsearch.org (Oct 21 2014)

      In this blog post, using a virtual machine sitting on the cloud, we’re going to show how to quickly set up a clustered instance of Elasticsearch to visualise firewall and honeypot datasources, namely IPtables and KippoSSH, focusing on the ELK-relevant configuration bits...

      (Read Full Article)

      Comment

    2. Mastering Security Analytics

      Explore Dark Reading (Oct 14 2014)

      When SIEM technology kicked off over a decade ago, the promise was that these platforms would become the catch-all system for storing and correlating security data across the enterprise to help analysts stop attacks in their tracks. But many SIEM platforms still can't pull in all of the necessary feeds to track attacks across the typical attack life cycle, or kill chain, which often spans endpoints, network resources, databases, and so on. Even when they can ingest data from, say, endpoint security systems, they are often unable to normalize it and pair it with related network security data that ...

      (Read Full Article)

      Comment Mentions:   Analytics

    3. The First-Ever Capability for Real-Time Analysis of Complex Anomalies in Big Data Sets

      Explore Yahoo! Finance (Oct 6 2014)

      Accurately identifying anomalous activities to detect the fingerprints of an advanced persistent threat or the cause of very complex IT performance issues requires a cross-correlated analysis of multiple data attributes. Performing this type of analysis at very large data scales has traditionally required a massive data transfer, which made real-time analysis impossible. By leveraging the statistical aggregation functions already available in platforms like Splunk and Elasticsearch, Prelert’s Stats Reduce overcomes this challenge...

      (Read Full Article)

      Comment Mentions:   Prelert   Big Data

    4. Keep Hackers at Bay the Security Analytics Way

      Explore Wipro (Oct 4 2014)

      A sophisticated hacking group recently attacked a public utility in the US and compromised its control system network. There has been a steep increase in such incidents globally, shifting the focus towards preventive mechanisms. Automation and pattern analysis are increasingly being used to counteract evolving threats. Statistics related to breaches in Critical Infrastructure bring out two important aspects...

      (Read Full Article)

      Comment Mentions:   Gartner   Root Cause   Analytics

  3. Recent Articles for Big Data in IT

    1. The American Business Awards: New Product Awards

      Explore Welcome to the Stevie Awards (Oct 8 2014)

      New Product or Service of the Year - Software - Big Data Solution (new category for 2014) 
      GOLD STEVIE WINNER:
      DataRPM, Fairfax, VA: DataRPM’s Instant Answers
      SILVER STEVIE WINNERS:
      Calabrio, Minneapolis, MN: Calabrio ONE® Speech Analytics
      Cloudera, Palo Alto, CA: Cloudera Enterprise 5
      DataRPM, Fairfax, VA: DataRPM
      MicroStrategy Inc., Tysons Corner, VA: MicroStrategy Analytics Desktop
      Prelert, Inc., Framingham, MA: Prelert, Inc.'s Anomaly Detective
      RSA, The Security Division of EMC, Bedford, MA: RSA Security Analytics
      Smarter Remarketer, Indianapolis, IN: Smarter Remarketer
      BRONZE STEVIE WINNERS:
      Agiliance, Sunnyvale, CA: Agiliance RiskVision 7
      Avention Inc., Concord, MA: Avention
      Internap, Atlanta, GA: Internap and Aerospike ...

      (Read Full Article)

      Comment

    2. ELK (Elasticsearch, Logstash and Kibana) Stack and Squid

      Explore Fernando Battistella (Oct 7 2014)

      Squid, the mean guy all users hate. Personally, as an admin, I hate it too, but its a necessary evil. Now, checking its logs is hell, whoever devised that timestamp surelly is a mean person. In time I will make a decent squid monitoring solution, its one of my professional goals, but for now I want a better way to look at those logs...

      (Read Full Article)

      Comment

    3. Security Analytics as a Services

      Explore Big Sonata (Sep 28 2014)

      Integrating Prelert’s anomaly detection engine into our big data platform creates a powerful combination of security analytics techniques, allowing us to identify unknown and advanced threats across petabytes of machine data we manage for our customers.” said Alert Logic’s Chief Strategy Officer, Misha Govshteyn.

      “Our objective has always been to help our customers respond to the most relevant security incidents before they impact their business. Working with Prelert allows us to leverage massive amounts of machine data we process every day to identify precursors to security breaches at the earliest possible moment and maintain our historically high degree ...

      (Read Full Article)

      Comment Mentions:   Prelert   Mark Jaffe   Analytics

    4. How Big Data security analytics is set to transform the security landscape - InformationWeek

      Explore InformationWeek (Sep 24 2014)

      To prevent emerging threats, security tools have to go beyond prevention and piece together different sets of information drawn from different events. For example, today, it is essential for event collection programs to go beyond firewall and IDS events, and add context.  “Identifying anomalous sequences of events at all layers of the stack is not enough. Understanding anomalous activity requires an understanding of the context — the “who, what and why”...

      (Read Full Article)

      Comment Mentions:   IBM   Big Data   Analytics

  4. Recent Articles for Machine Learning Analytics

    1. Making Sense of IoT Data With Machine Learning Technologies - Forbes

      Explore forbes.com (Sep 4 2014)

      As companies embark on the long journey of harvesting large amounts of data from connected devices and sensors, the valuable insights hidden in the data are driving up costs and not adding to the bottom line. How can these companies get these insights to market faster while reducing the risk of project failure? One way is to leverage the expertise of companies whose core competency is machine learning. One interesting use case comes from Prelert, a self-described anomaly detection company...

      (Read Full Article)

      Comment Mentions:   Prelert

    2. Sophie Chang Named VP of Engineering at Preler

      Explore businesswire.com (Aug 12 2014)

      Prelert , the anomaly detection company, today announced that it has hired Sophie Chang as Vice President of Engineering to lead its U.K.-based engineering team. In this role, Chang will be responsible for product development and managing all aspects of the team’s activities, helping to enhance Prelert’s machine learning-based anomaly detection engine. Chang brings more than ten years of senior executive experience to her new role, most notably through her time as VP Software at 1E, a fast-growing and successful B2B IT efficiency software company. She was responsible for growing its technology team from two people to ...

      (Read Full Article)

      Comment Mentions:   Prelert   Mark Jaffe   Dr. Steve

    3. How Machine Learning Is Improving Computer Security

      Explore smartdatacollective.com (Jul 27 2014)

      The machine learning approach has a major advantage over the more traditional way of threat detection. With the traditional way, systems had to look for signatures that had already been determined to be a threat. Once these signatures were identified within a network, the system would have to either stop it from further infiltration, or eliminate it. This method has some rather obvious weaknesses, the main one being its non-predictive nature. Machine learning is able to address this major weakness by looking through data for certain patterns and signals, thus predicting future attacks and preventing them, letting the system stay ...

      (Read Full Article)

      Comment

    4. Big data log analysis thrives on machine learning

      Explore infoworld.com (Jul 7 2014)

      Machine-generated log data is the dark matter of the big data cosmos. It is generated at every layer, node, and component within distributed information technology ecosystems, including smartphones and Internet-of-things endpoints. It is collected, processed, analyzed, and used everywhere, but mostly behind the scenes. Most of it is not designed or intended for direct human analysis. Unless filtered with brutal efficiency, the extreme volumes, velocities, and varieties of log data can quickly overwhelm human cognition. Clearly, automation is key to finding insights within log data, especially as it all scales into big data territory. Automation can ensure that data collection ...

      (Read Full Article)

      Comment

  5. Recent Articles

    1. Port Scan Intrusion Detection using ElasticSearch and Kibana

      Explore Linux Akademi (Sep 21 2014)

      One of the most fundamentals of security monitoring is to be aware of port scans which can be part of reconnaissance activity. Netflow is very critical in network situational awareness (NetSA), and utilizing Elastic Search and Kibana we can create ourselves a nice looking dashboard that makes it very easy to spot scanning activities...

      (Read Full Article)

      Comment

    2. Enterprise Annexation of Endpoint Security

      Explore Network World (Sep 11 2014)

      When it comes to strong cybersecurity, endpoints and servers have often been second-class citizens when compared to the network. I described this situation in a March 2013 blog post . According to ESG research, 58% of security professionals working at enterprise organizations (i.e. more than 1,000 employees) said that network security processes, skills, and technical controls were “much more thorough” or “somewhat more thorough” than server security processes, skills, and technical controls. This created a cybersecurity imbalance that can be easily exploited. Once cybercriminals and hackers navigate around the network, endpoints are sitting ducks for zero-day attacks, phishing scams ...

      (Read Full Article)

      Comment Mentions:   IBM   Cisco   Analytics

    3. Executing Aggregations

      Explore elasticsearch.org (Sep 1 2014)

      Aggregations provide the ability to group and extract statistics from your data. The easiest way to think about aggregations is by roughly equating it to the SQL GROUP BY and the SQL aggregate functions. In Elasticsearch, you have the ability to execute searches returning hits and at the same time return aggregated results separate from the hits all in one response. This is very powerful and efficient in the sense that you can run queries and multiple aggregations and get the results back of both (or either) operations in one shot avoiding network roundtrips using a concise and simplified API...

      (Read Full Article)

      Comment

    4. Logstash and Kibana via Docker

      Explore @ehazlett · @ehazlett (Aug 31 2014)

      With just a short while playing with the new Kibana and the editor, there is almost a limitless possibility of graphs, etc. for logging. Hopefully this is a quick way to test/deploy it yourself...

      (Read Full Article)

      Comment

  6. Recent Comments