1. Prelert Blog

    1. Occupy Your Data. Anomaly Detection Stops the Top 1% from Ruling IT.

      Explore Anomaly Detection Analytics (Aug 27 2014)

      Occupy Your Data. Anomaly Detection Stops the Top 1% from Ruling IT.

      How much of your data do you actually pay attention to?  Would you be surprised to realize it is probably far less than 1%?  How about 1% of 1%? This is the case in the vast majority of IT operations, performance management and security shops of any size anywhere in the world. But a typical web app involves hundreds if not thousands of components including software, networks, middleware, app servers, databases, etc. Now consider what happens when something breaks. Most of the time, one of the KPI you've selected triggers an alert or one of the dashboards you ...

      (Read Full Article)

      Comment Mentions:   Application Performance Management

    2. Data Exfiltration Detection via Behavioral Analysis

      Explore Anomaly Detection Analytics (Aug 21 2014)

      Data Exfiltration Detection via Behavioral Analysis

      There are many possible ways that one can detect “data exfiltration” (data theft), but in many cases, this involves either manual raw data inspection or the application of rules or signatures for specific behavioral violations. An alternative approach is to detect data exfiltration using automated behavioral anomaly detection using data that you’re probably already collecting and storing, and without the use of a DLP-specific security tool.

      (Read Full Article)

      Comment Mentions:   IT Security

    3. The Secret to Fixing Problems Before Users Find Them (part 2)

      Explore Anomaly Detection Analytics (Aug 19 2014)

      The Secret to Fixing Problems Before Users Find Them (part 2)

      In part 1 of this post , we talked about the failed paradigm of using thresholds and rules or 'eyeballs on timecharts' to monitor a critical app or service. In part 2 of this post we'll cover Anomaly Detection products that can leverage data you've already aggregated in stores like Splunk, Elasticsearch or NoSQL databases...

      (Read Full Article)

      Comment

    4. Choosing bucketSpan Wisely

      Explore Anomaly Detection Analytics (Aug 14 2014)

      Choosing bucketSpan Wisely

      In a previous blog post about optimizing the performance of the Engine API, I mentioned that choosing the proper bucketSpan results in not only a possible performance improvement, but I also alluded to bucketSpan affecting the timeliness and quality of your results. In effect, there is a 3-way balance between performance, timeliness of the results, and quality of the results that I’d like to dig into further here...

      (Read Full Article)

      Comment

    5. Static code analysis for C++

      Explore Anomaly Detection Analytics (Aug 12 2014)

      Static code analysis for C++

      Static code analysis has long been touted as a must have for high quality software. Unfortunately, my experience with it in previous jobs didn't live up to the hype. Within the last few years the majority of compilers have added a built-in static code analysis capability, so I thought it would be interesting to see how good they are...

      (Read Full Article)

      Comment

    6. Machine Learning, Anomaly Detection, and the Smart City

      Explore Anomaly Detection Analytics (Aug 7 2014)

      Machine Learning, Anomaly Detection, and the Smart City

      Burdened by heavy traffic, a major metropolitan city wanted to find a solution to help them improve travel times and congestion, and to minimize the effects of incidents and collisions on traffic. Since the city tracks accidents, events, construction, and other road problems, this kind of analysis can be done with automated anomaly detection. By analyzing the traffic and incident data, anomaly detection software can prioritize incidents so that problems with the most impact on traffic are addressed first...

      (Read Full Article)

      Comment Mentions:   Prelert

    7. The Secret to Fixing Problems Before Users Find Them (Part 1)

      Explore Anomaly Detection Analytics (Aug 4 2014)

      The Secret to Fixing Problems Before Users Find Them (Part 1)

      According to a TRAC Research survey on IT performance management challenges the top two issues were 1) 'Problems reported by end-users before IT finds them', and 2) 'too much time spent troubleshooting.' Despite crazy advances in every other field of IT technology, this problem really hasn't changed much in the last 20 years! The good news is we can show you how to change things through the following 3 incremental steps...

      (Read Full Article)

      Comment Mentions:   Google   Application Performance Management   Troubleshooting

    8. Tips to Optimize Performance with the Anomaly Detection Engine API

      Explore Anomaly Detection Analytics (Jul 31 2014)

      Tips to Optimize Performance with the Anomaly Detection Engine API

      As with any piece of software, there are performance considerations. If you’ve followed any of our developer blogs, you’ll quickly realize that Prelert’s engineers take creating high performance software seriously. But, performance is not only in how the software is architected, it is also in how you utilize the software. Here we will discuss some operational techniques that will optimize the performance of the Anomaly Detective Engine API...

      (Read Full Article)

      Comment Mentions:   Prelert

    9. The Unusually Common (Part 1): Methods for Modeling Categorical Data

      Explore Anomaly Detection Analytics (Jul 29 2014)

      The Unusually Common (Part 1): Methods for Modeling Categorical Data

      Recently, in the context of trying to understand how to quantify unusually common categories, I have found myself needing to study various properties of distributions on categorical data. Common building block distributions used to describe categorical data are the Bernoulli and categorical distributions. In fact, the Bernoulli is really just a special case of the categorical distribution with two categories. The categorical distribution is the distribution function of a random variable that takes one of categories m with probabilities { p i } = { p 1 , p 2 , . . . , p m } . The distributions I’m going to focus on are the result of counting ...

      (Read Full Article)

      Comment

    10. Expanding Data Analysis & Anomaly Detection Using partition & by Fields

      Explore Anomaly Detection Analytics (Jul 24 2014)

      Expanding Data Analysis & Anomaly Detection Using partition & by Fields

      In a previous blog, I showed how easy it is to analyze multiple metrics simultaneously by adding multiple “detectors” to your job configuration definition for the Anomaly Detective Engine API. Now, let’s take it a step further by expanding analysis across instances of things by using “byFieldName” and “partitionFieldName.”

      (Read Full Article)

      Comment

    11. Java Garbage Collectors: Comparing Performance Options

      Explore Anomaly Detection Analytics (Jul 23 2014)

      Java Garbage Collectors: Comparing Performance Options

      One of the benefits of Java over C++ is that memory management is handled by the JVM, liberating the developer from having to worry about it. At least, that’s the theory. In practice, when you write a non-trivial Java program you do have to consider what’s going on with the memory, but in a different way to writing C++. The JVM uses a garbage collector to find objects that are no longer required and release the memory they occupy. There are many ways that the Java garbage collector can be configured and no single optimum configuration: the most ...

      (Read Full Article)

      Comment

    12. Proud to be a Finalist: MassTLC Innovative Big Data Technology Award

      Explore Anomaly Detection Analytics (Jul 16 2014)

      Proud to be a Finalist: MassTLC Innovative Big Data Technology Award

      On July 15th at the Microsoft NERD Center , the Mass Technology Leadership Council (MassTLC) announced finalists for its 17th annual Leadership Awards , and we’re proud to let you know that Prelert was named a finalist in the “Innovative Technology of the Year – Big Data” category! With more than 550 member companies, MassTLC is the region’s leading technology association and the premier network for tech executives, entrepreneurs, investors and policy leaders. Prelert is being recognized for its Anomaly Detective software alongside other finalists in the category including EnerNOC, HP Vertica, Pixability and WordStream...

      (Read Full Article)

      Comment Mentions:   Prelert   Big Data

    13. Analyzing Multiple Metrics Using the Anomaly Detection Engine API

      Explore Anomaly Detection Analytics (Jul 15 2014)

      Analyzing Multiple Metrics Using the Anomaly Detection Engine API

      After hearing from some users that they assume it takes multiple jobs to analyze multiple metrics within the Anomaly Detective Engine API, I thought I’d write this blog to explain that analyzing multiple metrics within a single job is a lot easier to accomplish that you might think...

      (Read Full Article)

      Comment

    14. std::getline is the poor relation

      Explore Anomaly Detection Analytics (Jul 8 2014)

      std::getline is the poor relation

      Instead of using std::getline to read a line at a time from the file, this second program reads chunks of up to 256 bytes from the file, searches for newline characters using the std::find algorithm and then constructs a temporary string that can be moved onto the back of the vector or appends to the string already at the end of the vector if it was incomplete after the previous cycle.  There are all sorts of edge cases that have to be considered.  This is exactly the sort of code that puts people off programming in C++!

      (Read Full Article)

      Comment

    15. Connectors and Results Processors: Anomaly Detection Engine API

      Explore Anomaly Detection Analytics (Jul 3 2014)

      Connectors and Results Processors: Anomaly Detection Engine API

      From time to time, you might hear us reference the terms “Connector” or “Results Processor” in the context of integrating the Anomaly Detective Engine API into a data analysis workflow. So, what roles do each of these components play? The following diagram should help to put this into perspective...

      (Read Full Article)

      Comment Mentions:   Prelert   Hadoop

    16. Good Data for Anomaly Detection

      Explore Anomaly Detection Analytics (Jul 1 2014)

      Good Data for Anomaly Detection

      In general, the best type of data to use with the anomaly detection API engine is time stamped, structured data. This is data that is typically generated from machines (machine data), automated processes, sensor data, monitoring data, performance data, etc. This kind of data spans the disciplines of IT Operations, Application Performance Management, Security Logging/Detection, and more. Generally the data is temporal in nature, and its behavior over time is representative of how well (or not well) a system or a business process is working...

      (Read Full Article)

      Comment Mentions:   Application Performance Management   Big Data

    17. Data Science for the Rest of Us

      Explore Anomaly Detection Analytics (Jun 26 2014)

      Data Science for the Rest of Us

      First and foremost, the goal of Anomaly Detective ® is to bring sophisticated anomaly detection capabilities to the masses. Four kinds of anomalies satisfy a vast majority of use cases in IT Operations, Application Performance Management, Security, and other disciplines. They are: 1) Unexpected changes in the occurrence rate of things, 2) Unexpected changes in the values of things, 3) Rare things, and 4) Population/Peer Outliers. So if you use Splunk, the logical choice is to use Anomaly Detective for Splunk . If have some other data store, and if you're comfortable with the idea of scripting/coding against a ...

      (Read Full Article)

      Comment Mentions:   Prelert   Application Performance Management   Big Data

    18. Discrete Optimization in Machine Learning & Operations Research

      Explore Anomaly Detection Analytics (Jun 24 2014)

      Discrete Optimization in Machine Learning & Operations Research

      This post is going to look at optimization. Optimization is an extremely important problem. Two examples, which might be of particular interest to readers of this blog, are machine learning, which often boils down to optimizing some objective function, plus regularization term, for a set of observed and possibly labeled data, and operations research, which always contains an optimization component.

      (Read Full Article)

      Comment

  1. Recent Articles for IT Ops & APM

    1. The Enterprise in an App-Driven Universe

      Explore itbusinessedge.com (Aug 19 2014)

      While it’s tempting to simply throw more technology at application performance, the fact is that it is more than just an infrastructure problem. Management, automation, policy development and a host of other factors will have to be addressed and will need to be coordinated across on-premise, cloud, mobile and social media footprints in order to truly accommodate the diverse data environments that we now live in. But of all the changes that have hit the enterprise over the past decade, app performance is probably the most crucial because it gets to the heart of why the enterprise exists and ...

      (Read Full Article)

      Comment Mentions:   IBM   Application Performance Management   APM

    2. Network (In)Visibility Leads to IT Blame Game

      Explore APMdigest (Aug 19 2014)

      With more than half of US respondents (52%) confirming it costs their organization more than half a million dollars in revenue per hour when they have a network outage or performance degradation, you would assume that identifying unresolved network events would be a critical priority for IT organizations. This expectation is very much not the case – our survey revealed that 45% of organizations are still manually monitoring their networks...

      (Read Full Article)

      Comment Mentions:   Cloud Computing   Virtualization   BYOD

    3. Gartner Introduces the Criteria for the 2015 Network Performance Monitoring and Diagnostics (NPMD) Magic Quadrant

      Explore Gartner Blog Network (Aug 15 2014)

      NPMD tools allow for network engineers to understand the performance of applications and infrastructure components via network instrumentation. Additionally, these tools provide insight into the quality of the end user’s experience. The goal of NPMD products is not only to monitor the network components to facilitate outage and degradation resolution, but also to identify performance optimization opportunities. This is conducted via diagnostics, analytics and debugging capabilities to complement additional monitoring of today’s complex IT environments...

      (Read Full Article)

      Comment Mentions:   Gartner   Application Performance Monitoring   SaaS

    4. Monitoring Magic and the Future of APM

      Explore APMdigest (Aug 14 2014)

      APM has created an opportunity for itself to be absorbed into the mainstream of IT based on the value it provides. It's important to consider however, that with the abundance of monitoring tools available in the market, you don't buy APM, you develop it as a strategy, and then acquire the tools you need to realize the vision...

      (Read Full Article)

      Comment Mentions:   APM   Gartner

  2. Recent Articles for Security Analytics

    1. Understanding & Preventing Cyber Fraud & Attacks with Advanced Big Data Cyber Security Analytics

      Explore Dr. Chaos (Aug 18 2014)

      Some security professionals argue that it’s impossible to catch low-impact attacks, because attackers are getting smart about hiding their volume of attacks, essentially trying to fly under the radar and make their attacks look like legitimate traffic. This is often described as the needle in the haystack problem. How can you detect a malicious actor when there’s an overwhelming amount of other pieces of data that don’t mean anything? This is a lie. Data science experts will tell you that no matter how often an abnormal behavior occurs — whether it’s one hundred times or just once ...

      (Read Full Article)

      Comment Mentions:   Big Data   Analytics

    2. SIEM - The Next Frontier is Security Analytics

      Explore World's Largest Professional Network (Aug 18 2014)

      Security professional are overwhelmed by the sheer volume of notifications and alerts coming from both perimeter security systems and SIEMs; which unfortunately results in these alerts being ignored due to their lack of veracity. Forward thinking and aggressive security analysts are starting to understand through real-world events that in-order to protect their organization, they need to evolve from an all-alert methodology to defining risk patterns that allow what is perceived as a non-disparate security event is linked with sufficient critical asset information allowing a security executive and or security professional to react proactively not reactively...

      (Read Full Article)

      Comment Mentions:   Analytics

    3. Global security analytics market forecast to increase by 10.6% CAGR during 2013 to 2018

      Explore live-pr.com (Aug 18 2014)

      One of the major trends observed in the market is the growing adoption of SaaS-based security solutions. Since SaaS-based security solutions follow the pay-per-use model, they are more affordable than the traditional full license security solutions. In addition, these solutions provide several advantages such as ease of installation and upgradation, leading to their increased adoption...

      (Read Full Article)

      Comment Mentions:   EMC   SaaS   Analytics

    4. Predict, Prevent, and Act on Security Threats

      Explore IBM Data magazine (Aug 15 2014)

      Current approaches to network security can combat known threats, but they are not as good at finding new associations or uncovering patterns. As a result, organizations are opening the door to advanced persistent threats (APTs), spear phishing, hacktivism, and other dangers. Within all the noise of big data, organizations need sophisticated real-time analytics to find a relatively weak signal. Without deep insight, most threats cannot be detected. The goal is to predict, prevent, and act on threats to minimize damage, maintain a strong brand image, and keep employees, businesses, and information safe and secure...

      (Read Full Article)

      Comment Mentions:   IBM   InformationWeek   Big Data

  3. Recent Articles for Big Data in IT

    1. When Internet Of Things Meets Big Data

      Explore InformationWeek (Aug 25 2014)

      As always, the burden will fall on IT to solve the data storage, integration, and analytics dilemmas created by the IoT. You can't use your current strategy because the data to be captured, managed, and exploited will be even more diverse, and the use cases even more varied. IT's job will be to identify the best analytics platforms and tools to enable business users to acquire the data they need, analyze its meaning, and act on it quickly. While the range of analytics options available to accommodate big data in general seems broad, the number of systems that ...

      (Read Full Article)

      Comment Mentions:   Big Data   Analytics

    2. Machine Learning and Cognitive Systems, Part 2: Big Data Analytics

      Explore Innovation Insights (Aug 19 2014)

      In the context of analytics, and specifically Big Data analytics, the application of machine learning has a lot of potential for boosting the use of analytics to higher levels, and extend its use alongside other disciplines, such as artificial intelligence and cognition. But the applications need to be approached within the context of machine learning as enabler and enhancer, and must be integrated within an organizational analytics strategy...

      (Read Full Article)

      Comment Mentions:   Amazon   Big Data   Analytics

    3. Big Data Gets a Little More Manageable

      Explore itbusinessedge.com (Aug 15 2014)

      If you’d like to get up to snuff and are primarily interested in open source solutions, I recommend this CIOL.com column by Virenda Gupta , senior vice president at Huawei Technologies India. He discusses new open source solutions in the areas of Big Data processing, analytics and mining. He also addresses Big Data virtualization, where he sees a shortage of comprehensive platforms...

      (Read Full Article)

      Comment Mentions:   Google   Big Data   Hadoop

    4. What Big Data Doesn’t Do

      Explore Web Hosting Reviews & Ratings (Aug 14 2014)

      First and foremost, big data won’t solve your business problems. In fact no computer system will solve them. Given that the right questions are asked and right amount of thought, planning and execution goes into answering them, you can see what your problems are. Your skilled employees will use the data, analyze it, discuss it, come up with solutions, iterate, repeat and then solve these problems, not your big data analytics software...

      (Read Full Article)

      Comment Mentions:   Big Data   IDG   Analytics

  4. Recent Articles for Machine Learning Analytics

    1. Sophie Chang Named VP of Engineering at Preler

      Explore businesswire.com (Aug 12 2014)

      Prelert , the anomaly detection company, today announced that it has hired Sophie Chang as Vice President of Engineering to lead its U.K.-based engineering team. In this role, Chang will be responsible for product development and managing all aspects of the team’s activities, helping to enhance Prelert’s machine learning-based anomaly detection engine. Chang brings more than ten years of senior executive experience to her new role, most notably through her time as VP Software at 1E, a fast-growing and successful B2B IT efficiency software company. She was responsible for growing its technology team from two people to ...

      (Read Full Article)

      Comment Mentions:   Prelert   Mark Jaffe   Dr. Steve

    2. How Machine Learning Is Improving Computer Security

      Explore smartdatacollective.com (Jul 27 2014)

      The machine learning approach has a major advantage over the more traditional way of threat detection. With the traditional way, systems had to look for signatures that had already been determined to be a threat. Once these signatures were identified within a network, the system would have to either stop it from further infiltration, or eliminate it. This method has some rather obvious weaknesses, the main one being its non-predictive nature. Machine learning is able to address this major weakness by looking through data for certain patterns and signals, thus predicting future attacks and preventing them, letting the system stay ...

      (Read Full Article)

      Comment

    3. Big data log analysis thrives on machine learning

      Explore infoworld.com (Jul 7 2014)

      Machine-generated log data is the dark matter of the big data cosmos. It is generated at every layer, node, and component within distributed information technology ecosystems, including smartphones and Internet-of-things endpoints. It is collected, processed, analyzed, and used everywhere, but mostly behind the scenes. Most of it is not designed or intended for direct human analysis. Unless filtered with brutal efficiency, the extreme volumes, velocities, and varieties of log data can quickly overwhelm human cognition. Clearly, automation is key to finding insights within log data, especially as it all scales into big data territory. Automation can ensure that data collection ...

      (Read Full Article)

      Comment

    4. Prelert's new anomaly detection platform hopes to replace data scientists

      Explore Tech News (May 22 2014)

      If big data analytic player Prelert has its way, companies will be using its anomaly detection platform instead of turning to data scientists to make sense of it all. “We are putting anomaly detection capabilities into the hands of decision makers,” said Prelert chief executive Mark Jaffe. That is, instead of hiring a team of data scientists to run analyses on your data, Jaffe’s platform will do it for you. And protect it...

      (Read Full Article)

      Comment Mentions:   Prelert   Big Data   Mark Jaffe

  5. Recent Articles

    1. Installing Elastic Search on Amazon EC2 instance (Ubuntu 14.04)

      Explore Anoop Raveendran (Aug 17 2014)

      In this article I will explain how to set up an elastic server on EC2 instance. Before going to start we need to have an EC2 instance and I have an instance with Ubuntu 14.10 LTE installed. Normally we setup elastic search server in a separate EC2 instance for isolating the code base from the elastic search server...

      (Read Full Article)

      Comment Mentions:   Amazon   AWS

    2. A Look at Cyber Security Trends for 2014

      Explore smartdatacollective.com (Aug 14 2014)

      Security in particular, whether it’s network security, computer security , or IT security, is foremost on many business leaders’ minds. To prepare for what the future may hold, it’s important to look back at some of the recent trends to see the threats and solutions having the biggest impact on cyber security...

      (Read Full Article)

      Comment Mentions:   Cloud Security   Cloud Computing   Big Data

    3. Installing Logstash on a Windows server with Kibana in IIS

      Explore Ulyaoth (Aug 9 2014)

      This guide shows that it is also possible to run Logstash on a Windows machine and use IIS as web server...

      (Read Full Article)

      Comment

    4. Study Finds Security Analytics Significantly Decrease False Alerts

      Explore NewsLocker (Aug 6 2014)

      A study by EMA Research found that ninety percent of organizations who use security analytics have seen a decrease in false alerts or an improvement in actionable alerts by security personnel. Organizations that use security analytics/threat analytics are twice as likely to recover in minutes from unplanned incidents compared to those who don’t use analytics. Organizations who use security analytics/threat analytics are more than 50 percent more likely to have experienced reduced frequency and duration of investigations compared to those who don’t use analytics...

      (Read Full Article)

      Comment Mentions:   Analytics

  6. Recent Comments