1. Prelert Blog

    1. Machine Data is Different (and Why It Matters)

      Explore Anomaly Detection Analytics (Oct 13 2014)

      Machine Data is Different (and Why It Matters)

      Why is machine data different? To answer this question, let’s start by considering different perspectives on what constitutes unstructured data. In the case of a pre-defined set of allowed classifications, the standard approach for benchmarking a machine-learnt classification against a human-generated correct result is to use a confusion matrix . Each of the allowed classifications corresponds to a row and a column in the matrix, and the cells in the matrix record the number of input messages with corresponding human-generated classification and machine-determined classification. The perfect outcome is for all cells of the matrix to contain zeroes except those on ...

      (Read Full Article)

      Comment Mentions:   Prelert

    2. Ready to Talk Anomaly Detection & Advanced Math at Splunk's User Conference

      Explore Anomaly Detection Analytics (Oct 6 2014)

      Ready to Talk Anomaly Detection & Advanced Math at Splunk's User Conference

      The Prelert team, along with partners and customers, will share insights on using machine-based anomaly detection to find value in Big Data in front of over 4,000 IT and business professionals at Splunk’s fifth annual Worldwide Users’ Conference, .conf2014. The event will take place from October 6-9 at the MGM Grand in Las Vegas, Nevada.

      (Read Full Article)

      Comment Mentions:   Prelert   Big Data   Dr. Steve

    3. Anomaly Detection to Reduce the Noise

      Explore Anomaly Detection Analytics (Oct 2 2014)

      Anomaly Detection to Reduce the Noise

      If you have followed some of my other recent blogs, you’ll have noticed that automated anomaly detection is a great technique to find anomalous behaviors in data by effectively contrasting the difference between “normal” and “abnormal. " Most people equate this with contrasting between “good” and “bad,” but that isn’t always necessarily true. What if the data set you’re looking at are “all bad things,” such as Intrusion Detection (IDS) alerts?

      (Read Full Article)

      Comment Mentions:   IT Security

    4. Will You be Replaced by Machine Intelligence?

      Explore Anomaly Detection Analytics (Oct 1 2014)

      Will You be Replaced by Machine Intelligence?

      While humans are definitely needed for the expertise-dependent and creative functions, many aspects of IT operations and performance management could be done more effectively by machine intelligence. Here are just a few examples.

      Deciding What to Monitor
      Most application or service delivery environments have way more metrics, logs and event data than humans can reasonably get their heads around....

      Identifying Normal Behavior
      Even for the 1% of the data we do utilize, it is obvious that setting thresholds and alarm rules is a flawed approach...

      Finding Causal Relationships
      A recent survey by TRAC Research of APM pros turned up the ...

      (Read Full Article)

      Comment Mentions:   Application Performance Management   APM

    5. How to Detect (and Resolve) IT Ops/APM Issues Before Your Users Do

      Explore Anomaly Detection Analytics (Sep 26 2014)

      How to Detect (and Resolve) IT Ops/APM Issues Before Your Users Do

      As originally published by APMdigest. Among the most embarrassing situations for application support teams is first hearing about a critical performance issue from their users. With technology getting increasingly complex and IT environments changing almost overnight, the reality is that even the most experienced support teams are bound to miss a major problem with a critical application or service. One of the contributing factors is their continued reliance on traditional monitoring approaches.

      (Read Full Article)

      Comment Mentions:   APM

    6. Automated Anomaly Detection: A Connector for Amazon CloudWatch

      Explore Anomaly Detection Analytics (Sep 24 2014)

      Automated Anomaly Detection: A Connector for Amazon CloudWatch

      Amazon CloudWatch is a monitoring service for AWS cloud resources and the applications you run on AWS. You can use Amazon CloudWatch to collect and track metrics, collect and monitor log files, and set alarms. At the time of writing, CloudWatch is currently available to all AWS users, with the free version giving basic monitoring metrics (at 5 minute frequency) and generous usage limits. You can also add up to 10 custom metrics and 10 alarms. In this blog I shall explain why it is important to use unsupervised machine learning to effectively manage your AWS environments, and then point ...

      (Read Full Article)

      Comment Mentions:   AWS

    7. Rogue User Detection via Behavioral Analysis

      Explore Anomaly Detection Analytics (Sep 22 2014)

      Rogue User Detection via Behavioral Analysis

      Finding “rogue users” or “rogue systems” using behavioral analysis and automated anomaly detection takes a different approach than the traditional methods of manual data inspection, or the application of rules or signatures to identify specific behavioral violations. A “rogue” user or system, by definition, is someone or something that acts differently from the rest of the population. Therefore, using automated anomaly detection to find behavioral outliers via a comparison of users against each other (peer or behavioral analysis) is a viable approach...

      (Read Full Article)

      Comment Mentions:   IT Security

    8. Prelert Takes Home a Silver Stevie Award

      Explore Anomaly Detection Analytics (Sep 15 2014)

      Prelert Takes Home a Silver Stevie Award

      Last Friday marked the twelfth annual American Business Awards and Prelert was honored with a Silver Stevie Award in the New Product or Service of the Year - Software - Big Data Solution category. The announcement was made at the organization’s first ever New Product & Tech Awards banquet at the Palace Hotel in (where else but the tech mecca) San Francisco...

      (Read Full Article)

      Comment Mentions:   Prelert   Big Data

    9. It's Time to Democratize Data Science!

      Explore Anomaly Detection Analytics (Sep 11 2014)

      It's Time to Democratize Data Science!

      Can we realize our full potential by continuously improving advanced analytics that can only be used by data scientists? Is the right answer found in the 2013 prediction from a leading industry analyst that we need to focus our resources on educating millions of data scientists? There is no way that is sustainable. But there is an answer.  Data science can be packaged for the masses – and that is where our focus should be. Want to know how it's done?

      (Read Full Article)

      Comment Mentions:   Intel   Analytics

    10. Why What You Don't Know May Hurt You, & How Security Analytics Can Help

      Explore Anomaly Detection Analytics (Sep 10 2014)

      Why What You Don't Know May Hurt You, & How Security Analytics Can Help

      Attackers try hard to mask their activities and fly below the radar of your security paradigm – but try as they might, in order to accomplish their goals, their behaviors are going to have to be anomalous at some point in time. An authorized login is going to be attempted from a new IP address. A server is going to run a different process than usual. An unusual pattern of data transmissions will occur to a new external URL. The key to mitigating this threat is to be able to identify these ‘fingerprints’ amidst the billions of records produced by the ...

      (Read Full Article)

      Comment Mentions:   IT Security   Analytics

    11. How Security Analytics Help Identify and Manage Breaches

      Explore Anomaly Detection Analytics (Sep 3 2014)

      How Security Analytics Help Identify and Manage Breaches

      Statistical techniques are the only approach that can identify unknown attacks, and even when applied properly will still require a certain amount of human intervention. Security teams can definitely react a lot faster if they are immediately aware of previously unknown threats, so staying ahead of the bad guys really comes down to two things: the speed of a real-time analysis solution and the reaction time of the security team. In the end, this requires that both the right technology and organizational processes are in place...

      (Read Full Article)

      Comment Mentions:   IBM   Dr. Steve   IT Security

    12. Occupy Your Data. Anomaly Detection Stops the Top 1% from Ruling IT.

      Explore Anomaly Detection Analytics (Aug 27 2014)

      Occupy Your Data. Anomaly Detection Stops the Top 1% from Ruling IT.

      How much of your data do you actually pay attention to?  Would you be surprised to realize it is probably far less than 1%?  How about 1% of 1%? This is the case in the vast majority of IT operations, performance management and security shops of any size anywhere in the world. But a typical web app involves hundreds if not thousands of components including software, networks, middleware, app servers, databases, etc. Now consider what happens when something breaks. Most of the time, one of the KPI you've selected triggers an alert or one of the dashboards you ...

      (Read Full Article)

      Comment Mentions:   Application Performance Management

    13. Data Exfiltration Detection via Behavioral Analysis

      Explore Anomaly Detection Analytics (Aug 21 2014)

      Data Exfiltration Detection via Behavioral Analysis

      There are many possible ways that one can detect “data exfiltration” (data theft), but in many cases, this involves either manual raw data inspection or the application of rules or signatures for specific behavioral violations. An alternative approach is to detect data exfiltration using automated behavioral anomaly detection using data that you’re probably already collecting and storing, and without the use of a DLP-specific security tool.

      (Read Full Article)

      Comment Mentions:   IT Security

    14. The Secret to Fixing Problems Before Users Find Them (part 2)

      Explore Anomaly Detection Analytics (Aug 19 2014)

      The Secret to Fixing Problems Before Users Find Them (part 2)

      In part 1 of this post , we talked about the failed paradigm of using thresholds and rules or 'eyeballs on timecharts' to monitor a critical app or service. In part 2 of this post we'll cover Anomaly Detection products that can leverage data you've already aggregated in stores like Splunk, Elasticsearch or NoSQL databases...

      (Read Full Article)

      Comment

    15. Choosing bucketSpan Wisely

      Explore Anomaly Detection Analytics (Aug 14 2014)

      Choosing bucketSpan Wisely

      In a previous blog post about optimizing the performance of the Engine API, I mentioned that choosing the proper bucketSpan results in not only a possible performance improvement, but I also alluded to bucketSpan affecting the timeliness and quality of your results. In effect, there is a 3-way balance between performance, timeliness of the results, and quality of the results that I’d like to dig into further here...

      (Read Full Article)

      Comment

    16. Static code analysis for C++

      Explore Anomaly Detection Analytics (Aug 12 2014)

      Static code analysis for C++

      Static code analysis has long been touted as a must have for high quality software. Unfortunately, my experience with it in previous jobs didn't live up to the hype. Within the last few years the majority of compilers have added a built-in static code analysis capability, so I thought it would be interesting to see how good they are...

      (Read Full Article)

      Comment

    17. Machine Learning, Anomaly Detection, and the Smart City

      Explore Anomaly Detection Analytics (Aug 7 2014)

      Machine Learning, Anomaly Detection, and the Smart City

      Burdened by heavy traffic, a major metropolitan city wanted to find a solution to help them improve travel times and congestion, and to minimize the effects of incidents and collisions on traffic. Since the city tracks accidents, events, construction, and other road problems, this kind of analysis can be done with automated anomaly detection. By analyzing the traffic and incident data, anomaly detection software can prioritize incidents so that problems with the most impact on traffic are addressed first...

      (Read Full Article)

      Comment Mentions:   Prelert

    18. The Secret to Fixing Problems Before Users Find Them (Part 1)

      Explore Anomaly Detection Analytics (Aug 4 2014)

      The Secret to Fixing Problems Before Users Find Them (Part 1)

      According to a TRAC Research survey on IT performance management challenges the top two issues were 1) 'Problems reported by end-users before IT finds them', and 2) 'too much time spent troubleshooting.' Despite crazy advances in every other field of IT technology, this problem really hasn't changed much in the last 20 years! The good news is we can show you how to change things through the following 3 incremental steps...

      (Read Full Article)

      Comment Mentions:   Google   Application Performance Management   Troubleshooting

  1. Recent Articles for IT Ops & APM

    1. Simplified Big Data Analytics & Application Performance

      Explore Simplified Big Data Analytics (Oct 24 2014)

      With the evolution of big data, improved algorithms for search and correlation, smart dashboards/visualization and diagnostic capabilities, APM tools have matured to provide insights that you could never have before, thereby cutting troubleshooting time from days to minutes...

      (Read Full Article)

      Comment Mentions:   IBM   Google   Intel

    2. Differences Between Mobile and Server Performance Monitoring

      Explore APMdigest (Oct 15 2014)

      According to eMarketer, as of 2014 Americans consume more media using mobile devices than laptops and desktops combined. This shift in consumer behavior is also occurring within corporations, as employees increasingly rely on mobile devices for their work. With such a surge in mobile usage there is a growing need for corporations to ensure that their mobile experience is high quality and not broken. The following are 4 key differences that companies monitoring their server (and website) performance should consider when selecting a mobile app performance monitoring solution.

      (Read Full Article)

      Comment Mentions:   Application Performance Monitoring

    3. Deploying and migrating a multi-node ElasticSearch-Logstash-Kibana cluster using Docker

      Explore ClusterHQ (Oct 14 2014)

      ElasticSearch has exploded in popularity recently and it seems like almost everyone is using it for something, big or small . Running ElasticSearch in production, however, does not have a reputation of being easy. This post, shows how to use Flocker, an open-source data volume and container manager to deploy an ELK stack to multiple nodes, and then perform a near seamless database and data volume migration of ElasticSearch from one node to another...

      (Read Full Article)

      Comment

    4. Gartner: Top 10 Technology Trends for 2015 IT can’t ignore

      Explore Network World (Oct 7 2014)

      Gartner defines its Strategic Technology Trends as those technologies that have the most potential to drive great change in the enterprise IT arena in the next three years. Indeed this year’s crop has that potential...

      (Read Full Article)

      Comment Mentions:   IBM   Google   Microsoft

  2. Recent Articles for Security Analytics

    1. An Evolution Beyond Security Information & Event Management (SIEM)

      Explore Work-Bench (Oct 28 2014)

      Enterprises now realize that complete prevention of security incidents is impossible. Instead, there must be an increased focus on timely detection and response. Breaches WILL HAPPEN – so find them and contain them quickly. Both classic SIEM and Big Data approaches are compatible with this mindset and seek to unlock value through the aggregation and analysis of events generated by disparate systems. The problem is that SIEM promised the world but under delivered. Verizon’s 2013 Data Breach Investigations Report provides an indication of this, noting that only about 1% of data breaches were discovered through log review. This is due ...

      (Read Full Article)

      Comment Mentions:   Gartner   Big Data   Hadoop

    2. Big Data Security Analytics Landscape

      Explore Work-Bench (Oct 26 2014)

      Big Data Security Analytics is an emerging market and we’re certainly excited to see how it evolves. In the short-term, we’re interested to know which companies we missed. We’ll also be monitoring how well these companies gain traction with enterprise customers. Longer-term, it will be interesting to see which approaches prove most effective at detecting and preventing attackers. 

      (Read Full Article)

      Comment Mentions:   IBM   Gartner   Big Data

    3. Use ELK To Visualise Security Data

      Explore elasticsearch.org (Oct 21 2014)

      In this blog post, using a virtual machine sitting on the cloud, we’re going to show how to quickly set up a clustered instance of Elasticsearch to visualise firewall and honeypot datasources, namely IPtables and KippoSSH, focusing on the ELK-relevant configuration bits...

      (Read Full Article)

      Comment

    4. Cyber Threat Monitoring System with Ossec + ZeroMQ + Logstash + ElasticSearch and Kibana

      Explore Mehmet INCE (Oct 20 2014)

      I’ve been using Ossec as Intrusion Detection System for year. Ossec is awesome service for detection and notification. Thus I’ve decided to build a cyber threat monitoring system with open source technologies. In order to do that, I decided to get logs from Ossec and send them to the Elasticsearch engine. This write-up going to be about installation of Ossec and Logstash/ElastichSearch and integration between of these services.

      (Read Full Article)

      Comment

  3. Recent Articles for Big Data in IT

    1. Gartner: 8 big trends in big data analytics

      Explore Computerworld (Oct 23 2014)

      "IT managers and implementers cannot use lack of maturity as an excuse to halt experimentation," says Mark Beyer, an analyst at Gartner. Initially, only a few people -- the most skilled analysts and data scientists -- need to experiment. Then those advanced users and IT should jointly determine when to deliver new resources to the rest of the organization. And IT shouldn't necessarily rein in analysts who want to move ahead full-throttle. Rather, Beyer says, IT needs to work with analysts to "put a variable-speed throttle on these new high-powered tools."

      (Read Full Article)

      Comment Mentions:   IBM   Google   Amazon

    2. Security Analytics in Action: Use Cases for Deep Monitoring of Privileged Users

      Explore Pivotal P.O.V. (Oct 13 2014)

      In this post, we will address the problem of detecting privilege misuse from help desk administrators and show you how data science is applied in monitoring activities with anomaly alerting...

      (Read Full Article)

      Comment Mentions:   Analytics

    3. The American Business Awards: New Product Awards

      Explore Welcome to the Stevie Awards (Oct 8 2014)

      New Product or Service of the Year - Software - Big Data Solution (new category for 2014) 
      GOLD STEVIE WINNER:
      DataRPM, Fairfax, VA: DataRPM’s Instant Answers
      SILVER STEVIE WINNERS:
      Calabrio, Minneapolis, MN: Calabrio ONE® Speech Analytics
      Cloudera, Palo Alto, CA: Cloudera Enterprise 5
      DataRPM, Fairfax, VA: DataRPM
      MicroStrategy Inc., Tysons Corner, VA: MicroStrategy Analytics Desktop
      Prelert, Inc., Framingham, MA: Prelert, Inc.'s Anomaly Detective
      RSA, The Security Division of EMC, Bedford, MA: RSA Security Analytics
      Smarter Remarketer, Indianapolis, IN: Smarter Remarketer
      BRONZE STEVIE WINNERS:
      Agiliance, Sunnyvale, CA: Agiliance RiskVision 7
      Avention Inc., Concord, MA: Avention
      Internap, Atlanta, GA: Internap and Aerospike ...

      (Read Full Article)

      Comment

    4. ELK (Elasticsearch, Logstash and Kibana) Stack and Squid

      Explore Fernando Battistella (Oct 7 2014)

      Squid, the mean guy all users hate. Personally, as an admin, I hate it too, but its a necessary evil. Now, checking its logs is hell, whoever devised that timestamp surelly is a mean person. In time I will make a decent squid monitoring solution, its one of my professional goals, but for now I want a better way to look at those logs...

      (Read Full Article)

      Comment

  4. Recent Articles for Machine Learning Analytics

    1. Making Sense of IoT Data With Machine Learning Technologies - Forbes

      Explore forbes.com (Sep 4 2014)

      As companies embark on the long journey of harvesting large amounts of data from connected devices and sensors, the valuable insights hidden in the data are driving up costs and not adding to the bottom line. How can these companies get these insights to market faster while reducing the risk of project failure? One way is to leverage the expertise of companies whose core competency is machine learning. One interesting use case comes from Prelert, a self-described anomaly detection company...

      (Read Full Article)

      Comment Mentions:   Prelert

    2. Sophie Chang Named VP of Engineering at Preler

      Explore businesswire.com (Aug 12 2014)

      Prelert , the anomaly detection company, today announced that it has hired Sophie Chang as Vice President of Engineering to lead its U.K.-based engineering team. In this role, Chang will be responsible for product development and managing all aspects of the team’s activities, helping to enhance Prelert’s machine learning-based anomaly detection engine. Chang brings more than ten years of senior executive experience to her new role, most notably through her time as VP Software at 1E, a fast-growing and successful B2B IT efficiency software company. She was responsible for growing its technology team from two people to ...

      (Read Full Article)

      Comment Mentions:   Prelert   Mark Jaffe   Dr. Steve

    3. How Machine Learning Is Improving Computer Security

      Explore smartdatacollective.com (Jul 27 2014)

      The machine learning approach has a major advantage over the more traditional way of threat detection. With the traditional way, systems had to look for signatures that had already been determined to be a threat. Once these signatures were identified within a network, the system would have to either stop it from further infiltration, or eliminate it. This method has some rather obvious weaknesses, the main one being its non-predictive nature. Machine learning is able to address this major weakness by looking through data for certain patterns and signals, thus predicting future attacks and preventing them, letting the system stay ...

      (Read Full Article)

      Comment

    4. Big data log analysis thrives on machine learning

      Explore infoworld.com (Jul 7 2014)

      Machine-generated log data is the dark matter of the big data cosmos. It is generated at every layer, node, and component within distributed information technology ecosystems, including smartphones and Internet-of-things endpoints. It is collected, processed, analyzed, and used everywhere, but mostly behind the scenes. Most of it is not designed or intended for direct human analysis. Unless filtered with brutal efficiency, the extreme volumes, velocities, and varieties of log data can quickly overwhelm human cognition. Clearly, automation is key to finding insights within log data, especially as it all scales into big data territory. Automation can ensure that data collection ...

      (Read Full Article)

      Comment

  5. Recent Articles

    1. Mastering Security Analytics

      Explore Dark Reading (Oct 14 2014)

      When SIEM technology kicked off over a decade ago, the promise was that these platforms would become the catch-all system for storing and correlating security data across the enterprise to help analysts stop attacks in their tracks. But many SIEM platforms still can't pull in all of the necessary feeds to track attacks across the typical attack life cycle, or kill chain, which often spans endpoints, network resources, databases, and so on. Even when they can ingest data from, say, endpoint security systems, they are often unable to normalize it and pair it with related network security data that ...

      (Read Full Article)

      Comment Mentions:   Analytics

    2. How we switched elasticsearch clusters without anybody noticing

      Explore Reverb.com Dev Blog (Oct 8 2014)

      We’ve been stuck on an old (0.90) version of elasticsearch for some time now, looking for a way to upgrade to 1.3 so we can get all the benefits of the scalable percolator and other fun features. We’ve recently been working on some interesting Feed functionality that lets you follow searches on the site and be alerted when new items are posted, and we needed the scalability of the percolator in 1.3 to make this work smoothly. The challenge was switching from a single 0.90 node to a cluster of 1.3 nodes on ...

      (Read Full Article)

      Comment

    3. Keep Hackers at Bay the Security Analytics Way

      Explore Wipro (Oct 4 2014)

      A sophisticated hacking group recently attacked a public utility in the US and compromised its control system network. There has been a steep increase in such incidents globally, shifting the focus towards preventive mechanisms. Automation and pattern analysis are increasingly being used to counteract evolving threats. Statistics related to breaches in Critical Infrastructure bring out two important aspects...

      (Read Full Article)

      Comment Mentions:   Gartner   Root Cause   Analytics

    4. Port Scan Intrusion Detection using ElasticSearch and Kibana

      Explore Linux Akademi (Sep 21 2014)

      One of the most fundamentals of security monitoring is to be aware of port scans which can be part of reconnaissance activity. Netflow is very critical in network situational awareness (NetSA), and utilizing Elastic Search and Kibana we can create ourselves a nice looking dashboard that makes it very easy to spot scanning activities...

      (Read Full Article)

      Comment

  6. Recent Comments