SlideShare a Scribd company logo
1 of 44
Download to read offline
Dynamic Lookups
Agenda

Lookups in General

Static Lookups

Dynamic Lookups
 -   Retrieve fields from a web site
 -   Retrieve fields from a database
 -   Retrieve fields from a persistent cache

                          2
Enrich Your Events with Fields from External Sources




                         3
Splunk: The Engine for Machine Data

   Customer                                                                                                                 Outside the
  Facing Data                                                                                                               Datacenter
Click-stream data                                                                                                        Manufacturing, logistics
Shopping cart data                                                                                                       …
Online transaction data                                                                                                  CDRs & IPDRs
                                                                                                                         Power consumption
                              Logfiles      Configs Messages   Traps        Metrics   Scripts    Changes    Tickets      RFID data
                                                               Alerts                                                    GPS data


                                                    Virtualization
   Windows                 Linux/Unix                                          Applications                Databases        Networking
                                                       & Cloud
 Registry                 Configurations            Hypervisor                Web logs                Configurations      Configurations
 Event logs               syslog                    Guest OS, Apps            Log4J, JMS, JMX         Audit/query logs    syslog
 File system              File system               Cloud                     .NET events             Tables              SNMP
 sysinternals             ps, iostat, top                                     Code and scripts        Schemas             netflow



                                                                        4
5
6
7
8
Interesting Things to Lookup


•   User’s Mailing Address          •   External Host Address
•   Error Code Descriptions         •   Database Query
•   Product Names                   •   Web Service Call for Status
•   Stock Symbol (from CUSIP)       •   Geo Location




                                9
Other Reasons For Lookup
• Bypass static developer or vendor that does not enrich logs
• Imaginative correlations
   • Example: A website URL with “Like” or “Dislike” count
     stored in external source
• Make your data more interesting
   • Better to see textual descriptions than arcane codes



                               10
Agenda

Lookups in General

Static Lookups

Dynamic Lookups
 -   Retrieve fields from a web site
 -   Retrieve fields from a database
 -   Retrieve fields from a persistent cache

                          11
Static vs. Dynamic Lookup


                         External Data comes from a CSV file
 Static



Dynamic
              External Data comes from output of external script, which
                                resembles a CSV file




                            12
Static Lookup Review
• Pick the input fields that will be used to get output fields
• Create or locate a CSV file that has all the fields you need in the
  proper order
• Tell Splunk via the Manager about your CSV file and your lookup
   • You can also define lookups manually via props.conf and
      transforms.conf
   • If you use automatic lookups, they will run every time the
      source, sourcetype or associated host stanza is used in a search
   • Non-automatic lookups run only when the lookup command is
      invoked in the search
                                   13
Example Static Lookup Conf Files
props.conf
         [access_combined]

         lookup_http = http_status status
                OUTPUT status_description, status_type
transforms.conf
         [http_status]


         filename = http_status.csv


                             14
Permissions
Define Lookups via Splunk Manager & set permissions there
                        local.meta

    [lookups/http_status.csv]
    export = system

    [transforms/http_status]
    export = system



                                15
Example Automatic Static Lookup




               16
Agenda

Lookups in General

Static Lookups

Dynamic Lookups
 -   Retrieve fields from a web site
 -   Retrieve fields from a database
 -   Retrieve fields from a persistent cache

                          17
Dynamic Lookups

• Write the script to simulate access to external source
• Test the script with one set of inputs
• Create the Splunk Version of the lookup script
• Register the script with Splunk via Manager or conf files
• Test the script explicitly before using automatic lookups



                              18
Lookups vs Custom Command
• Use dynamic lookups when returning fields given input fields
   • Standard use case for users who already are familiar with lookups
• Use a custom command when doing MORE than a lookup
   • Not all use cases involve just returning fields
       • Decrypt event data
       • Translate event data from one format to another with new fields
          (e.g. FIX)


                                     19
Write/Test External Field Gathering Script


                    Send: Input Fields
External Data in
Cloud                                      Your Python Script
                   Return: Output Fields




                          20
Example Script to Test External Lookup

# Given a host, find the corresponding IP address
def mylookup(host):
  try:
    ipaddrlist = socket.gethostbyname_ex(host)
    return ipaddrlist
  except:
  return[]

                        21
External Field Gathering Script with Splunk



External Data in
Cloud                    Your Python Script
                                        Return: Output Fields




                          22
Script for Splunk Simulates Reading Input CSV

          hostname, ip

          a.b.c.com

          zorrosty.com

          seemanny.com



                         23
Output of Script Returns Logically Complete CSV

           hostname, ip

           a.b.c.com, 1.2.3.4

           zorrosty.com, 192.168.1.10

           seemanny.com, 10.10.2.10



                          24
transforms.conf for Dynamic Lookup

[NameofLookup]
external_cmd = <name>.py field1….fieldN
external_type = python
fields_list = field1, …, fieldN




                        25
Example Dynamic Lookup conf files

             transforms.conf
   # Note – this is an explicit lookup

   [whoisLookup]
   external_cmd = whois_lookup.py ip whois
   external_type = python
   fields_list = ip, whois



                    26
Dynamic Lookup Python Flow
def lookup(input):
  Perform external lookup based on input. Return result

main()
Check standard input for CSV headers.

Write headers to standard output.

For each line in standard input (input fields):
 Gather input fields into a dictionary (key-value structure)
 ret = lookup(input fields)
 If ret:
 Send to standard output input values and return values
    from lookup

                                           27
Whois Lookup
def main():
  if len(sys.arv) != 3:
     print “Usage: python whois_lookup.py [ip field] [whois field]”
  sys.exit(0)
  ipf = sys.argv[1]
 whoisf = sys.argv[2]
 r = csv.reader(sys.stdin)
 w = none
 header = [ ]
 first = True…


                                        28
Whois Lookup (cont.) to Read CSV Header
# First get read the “CSV Header” and output the field names
for line in r:
  if first:
      header = line
      if whoisf not in header or ipf not in header:
         print “IP and whois fields must exist in CSV data”
         sys.exit(0)
      csv.write(sys.stdout).writerow(header)
      w = csv.DictWriter(sys.stdout, header)
      first = False
     continue…

                                    29
Whois Lookup (cont.) to Populate Input Fields
# Read the result and populate the values for the input fields (ip
address in our case)
    result = {}
    i=0
    while i < len(header):
      if i < len(line):
          result[header[i]] = line[i]
      else:
          result[header[i]] = ''
      i += 1

                                  30
Whois Lookup (cont.) to Populate Input Fields
# Perform the whois lookup if necessary
     if len(result[ipf]) and len(result[whoisf]):
         w.writerow(result)
# Else call external website to get whois field from the ip address as the
key
     elif len(result[ipf]):
         result[whoisf] = lookup(result[ipf])
         if len(result[whoisf]):
             w.writerow(result)


                                    31
Whois Lookup Function
LOCATION_URL=http://some.url.com?query=
# Given an ip, return the whois response
def lookup(ip):
  try:
      whois_ret = urllib.urlopen(LOCATION_URL + ip)
      lines = whois_ret.readlines()
      return lines
  except:
      return ''


                                    32
Database Lookup

• Acquire proper modules to connect to the database
• Connect and authenticate to database
   • Use a connection pool if possible
• Have lookup function query the database
   • Return a list([]) of results



                            33
Database Lookup vs. Database Sent To Index
• Well, it depends…
• Use a Lookup when:
   • Using needle in the haystack searches with a few users
   • Using form searches returning few results
• Index the database table or view when:
   • Having LOTS of users and ad hoc reporting is needed
   • It’s OK to have “stale” data (N minutes) old for a dynamic
     database

                                34
Example Database Lookup using MySQL

# First connect to DB outside of the for loop

conn = MySQLdb.connect(host = “localhost”,
                                 user = “name of user”,
                                 passwd = “password”,
                                 db = “Name of DB”)

cursor = conn.cursor()



                                 35
Example Database Lookup (cont.) using MySQL
import MySQLdb…

# Given a city, find its country

def lookup(city, cur):
 try:
    selString=“SELECT country FROM city_country where city=“
    cur.execute(selString + “”” + city + “””)
    row = cur.fetechone()
    return row[0]
 except:
    return []


                                       36
Lookup Using Key Value Persistent Cache

• Download and install Redis
• Download and install Redis Python module
                                                  Redis is an open
• Import Redis module in Python and populate      source, advanced key-
                                                  value store.
  key value DB
• Import Redis module in lookup function
  given to Splunk to lookup a value given a key


                                37
Redis Lookup
###CHANGE PATH According to your REDIS install ######
sys.path.append(“/Library/Python/2.6/…/redis-2.4.5-py.egg”)
import redis
…
def main()
…
#Connect to redis – Change for your distribution
pool = redis.ConnectionPool(host=„localhost‟,port=6379,db=0)
redp = redis.Redis(connection_pool=pool)




                                         38
Redis Lookup (cont.)

def lookup(redp, mykey):

try:
  return redp.get(mykey)

except:
  return “”




                 39
Combine Persistent Cache with External Lookup
• For data that is “relatively static”
   • First see if the data is in the persistent cache
   • If not, look it up in the external source such as a database or
     web service
   • If results come back, add results to the persistent cache and
     return results
• For data that changes often, you will need to create your own cache
  retention policies

                                 40
Combining Redis with Whois Lookup
def lookup(redp, ip):
  try:
      ret = redp.get(ip)
      if ret!=None and ret!='':
          return ret
      else:
          whois_ret = urllib.urlopen(LOCATION_URL + ip)
          lines = whois_ret.readlines()
          if lines!='':
               redp.set(ip, lines)
          return lines…
  except:


                                    41
Where do I get the add-ons from today?
                            Splunkbase!
     Add-On                       Download Location                    Release

                   http://splunk-base.splunk.com/apps/22381/whois-   4.x
     Whois         add-on

                   http://splunk-                                    4.x
    DBLookup       base.splunk.com/apps/22394/example-lookup-
                   using-a-database
                   http://splunk-base.splunk.com/apps/27106/redis-   4.x
  Redis Lookup     lookup

                   http://splunk-base.splunk.com/apps/22282/geo-     4.x
Geo IP Lookup (not
                   location-lookup-script-powered-by-maxmind
 in these slides)
                                        42
Conclusion


Lookups are a powerful way to enhance
your search experience beyond indexing
               the data.


                   43
Thank You

More Related Content

What's hot

Unit IV UNCERTAINITY AND STATISTICAL REASONING in AI K.Sundar,AP/CSE,VEC
Unit IV UNCERTAINITY AND STATISTICAL REASONING in AI K.Sundar,AP/CSE,VECUnit IV UNCERTAINITY AND STATISTICAL REASONING in AI K.Sundar,AP/CSE,VEC
Unit IV UNCERTAINITY AND STATISTICAL REASONING in AI K.Sundar,AP/CSE,VECsundarKanagaraj1
 
Decision Tree Learning
Decision Tree LearningDecision Tree Learning
Decision Tree LearningMilind Gokhale
 
Heuristic Search Techniques {Artificial Intelligence}
Heuristic Search Techniques {Artificial Intelligence}Heuristic Search Techniques {Artificial Intelligence}
Heuristic Search Techniques {Artificial Intelligence}FellowBuddy.com
 
Machine learning
Machine learningMachine learning
Machine learningAmit Rathi
 
Issues in knowledge representation
Issues in knowledge representationIssues in knowledge representation
Issues in knowledge representationSravanthi Emani
 
Case study on deep learning
Case study on deep learningCase study on deep learning
Case study on deep learningHarshitBarde
 
Supervised Machine Learning in R
Supervised  Machine Learning  in RSupervised  Machine Learning  in R
Supervised Machine Learning in RBabu Priyavrat
 
High level design document template
High level design document templateHigh level design document template
High level design document templateanosha jamshed
 
A review of machine learning based anomaly detection
A review of machine learning based anomaly detectionA review of machine learning based anomaly detection
A review of machine learning based anomaly detectionMohamed Elfadly
 
Ensemble learning
Ensemble learningEnsemble learning
Ensemble learningHaris Jamil
 
Random Forest Algorithm - Random Forest Explained | Random Forest In Machine ...
Random Forest Algorithm - Random Forest Explained | Random Forest In Machine ...Random Forest Algorithm - Random Forest Explained | Random Forest In Machine ...
Random Forest Algorithm - Random Forest Explained | Random Forest In Machine ...Simplilearn
 
Lecture 4 Decision Trees (2): Entropy, Information Gain, Gain Ratio
Lecture 4 Decision Trees (2): Entropy, Information Gain, Gain RatioLecture 4 Decision Trees (2): Entropy, Information Gain, Gain Ratio
Lecture 4 Decision Trees (2): Entropy, Information Gain, Gain RatioMarina Santini
 
Reinforcement learning
Reinforcement learning Reinforcement learning
Reinforcement learning Chandra Meena
 

What's hot (20)

Random forest
Random forestRandom forest
Random forest
 
Unit IV UNCERTAINITY AND STATISTICAL REASONING in AI K.Sundar,AP/CSE,VEC
Unit IV UNCERTAINITY AND STATISTICAL REASONING in AI K.Sundar,AP/CSE,VECUnit IV UNCERTAINITY AND STATISTICAL REASONING in AI K.Sundar,AP/CSE,VEC
Unit IV UNCERTAINITY AND STATISTICAL REASONING in AI K.Sundar,AP/CSE,VEC
 
Naive bayes
Naive bayesNaive bayes
Naive bayes
 
Decision Tree Learning
Decision Tree LearningDecision Tree Learning
Decision Tree Learning
 
Heuristic Search Techniques {Artificial Intelligence}
Heuristic Search Techniques {Artificial Intelligence}Heuristic Search Techniques {Artificial Intelligence}
Heuristic Search Techniques {Artificial Intelligence}
 
Machine learning
Machine learningMachine learning
Machine learning
 
Issues in knowledge representation
Issues in knowledge representationIssues in knowledge representation
Issues in knowledge representation
 
Registry forensics
Registry forensicsRegistry forensics
Registry forensics
 
Case study on deep learning
Case study on deep learningCase study on deep learning
Case study on deep learning
 
Supervised Machine Learning in R
Supervised  Machine Learning  in RSupervised  Machine Learning  in R
Supervised Machine Learning in R
 
High level design document template
High level design document templateHigh level design document template
High level design document template
 
A review of machine learning based anomaly detection
A review of machine learning based anomaly detectionA review of machine learning based anomaly detection
A review of machine learning based anomaly detection
 
Ensemble learning
Ensemble learningEnsemble learning
Ensemble learning
 
Random Forest Algorithm - Random Forest Explained | Random Forest In Machine ...
Random Forest Algorithm - Random Forest Explained | Random Forest In Machine ...Random Forest Algorithm - Random Forest Explained | Random Forest In Machine ...
Random Forest Algorithm - Random Forest Explained | Random Forest In Machine ...
 
Big data ppt
Big data pptBig data ppt
Big data ppt
 
CS8080 IRT UNIT I NOTES.pdf
CS8080 IRT UNIT I  NOTES.pdfCS8080 IRT UNIT I  NOTES.pdf
CS8080 IRT UNIT I NOTES.pdf
 
Data Cleaning Techniques
Data Cleaning TechniquesData Cleaning Techniques
Data Cleaning Techniques
 
Firewall Basing
Firewall BasingFirewall Basing
Firewall Basing
 
Lecture 4 Decision Trees (2): Entropy, Information Gain, Gain Ratio
Lecture 4 Decision Trees (2): Entropy, Information Gain, Gain RatioLecture 4 Decision Trees (2): Entropy, Information Gain, Gain Ratio
Lecture 4 Decision Trees (2): Entropy, Information Gain, Gain Ratio
 
Reinforcement learning
Reinforcement learning Reinforcement learning
Reinforcement learning
 

Viewers also liked

Data Visualization on the Tech Side
Data Visualization on the Tech SideData Visualization on the Tech Side
Data Visualization on the Tech SideMathieu Elie
 
Opensource approach to design and deployment of Microservices based VNF
Opensource approach to design and deployment of Microservices based VNFOpensource approach to design and deployment of Microservices based VNF
Opensource approach to design and deployment of Microservices based VNFMichelle Holley
 
Journey of The Connected Enterprise - Knowledge Graphs - Smart Data
Journey of The Connected Enterprise - Knowledge Graphs - Smart DataJourney of The Connected Enterprise - Knowledge Graphs - Smart Data
Journey of The Connected Enterprise - Knowledge Graphs - Smart DataBenjamin Nussbaum
 
Using a Canary Microservice to Validate the Software Delivery Pipeline
Using a Canary Microservice to Validate the Software Delivery PipelineUsing a Canary Microservice to Validate the Software Delivery Pipeline
Using a Canary Microservice to Validate the Software Delivery PipelineXebiaLabs
 
Turnkey Riak KV Cluster
Turnkey Riak KV ClusterTurnkey Riak KV Cluster
Turnkey Riak KV ClusterJoe Olson
 
Demystifying Security Analytics: Data, Methods, Use Cases
Demystifying Security Analytics: Data, Methods, Use CasesDemystifying Security Analytics: Data, Methods, Use Cases
Demystifying Security Analytics: Data, Methods, Use CasesPriyanka Aash
 
Micro Services - Small is Beautiful
Micro Services - Small is BeautifulMicro Services - Small is Beautiful
Micro Services - Small is BeautifulEberhard Wolff
 
Tubular Labs - Using Elastic to Search Over 2.5B Videos
Tubular Labs - Using Elastic to Search Over 2.5B VideosTubular Labs - Using Elastic to Search Over 2.5B Videos
Tubular Labs - Using Elastic to Search Over 2.5B VideosTubular Labs
 
Heterogenous Persistence
Heterogenous PersistenceHeterogenous Persistence
Heterogenous PersistenceJervin Real
 
Combining sentences with the words although and despite
Combining sentences with the words although and despiteCombining sentences with the words although and despite
Combining sentences with the words although and despiteEmily Kissner
 
Bsides Delhi Security Automation for Red and Blue Teams
Bsides Delhi Security Automation for Red and Blue TeamsBsides Delhi Security Automation for Red and Blue Teams
Bsides Delhi Security Automation for Red and Blue TeamsSuraj Pratap
 
Docker in Production, Look No Hands! by Scott Coulton
Docker in Production, Look No Hands! by Scott CoultonDocker in Production, Look No Hands! by Scott Coulton
Docker in Production, Look No Hands! by Scott CoultonDocker, Inc.
 
Deploying services: automation with docker and ansible
Deploying services: automation with docker and ansibleDeploying services: automation with docker and ansible
Deploying services: automation with docker and ansibleJohn Zaccone
 
Adaptive Content Show & Tell - Austin Content
Adaptive Content Show & Tell - Austin ContentAdaptive Content Show & Tell - Austin Content
Adaptive Content Show & Tell - Austin Contentcdelk
 
Build a Configurable Enterprise SaaS App in Only 9 Months
Build a Configurable Enterprise SaaS App in Only 9 MonthsBuild a Configurable Enterprise SaaS App in Only 9 Months
Build a Configurable Enterprise SaaS App in Only 9 MonthsMongoDB
 
SocCnx11 - All you need to know about orient me
SocCnx11 - All you need to know about orient meSocCnx11 - All you need to know about orient me
SocCnx11 - All you need to know about orient mepanagenda
 
Secure Yourself, Practice what we preach - BSides Austin 2015
Secure Yourself, Practice what we preach - BSides Austin 2015Secure Yourself, Practice what we preach - BSides Austin 2015
Secure Yourself, Practice what we preach - BSides Austin 2015Michael Gough
 
Accelerated Leadership
Accelerated LeadershipAccelerated Leadership
Accelerated Leadershipkktv
 

Viewers also liked (20)

Data Visualization on the Tech Side
Data Visualization on the Tech SideData Visualization on the Tech Side
Data Visualization on the Tech Side
 
Opensource approach to design and deployment of Microservices based VNF
Opensource approach to design and deployment of Microservices based VNFOpensource approach to design and deployment of Microservices based VNF
Opensource approach to design and deployment of Microservices based VNF
 
Journey of The Connected Enterprise - Knowledge Graphs - Smart Data
Journey of The Connected Enterprise - Knowledge Graphs - Smart DataJourney of The Connected Enterprise - Knowledge Graphs - Smart Data
Journey of The Connected Enterprise - Knowledge Graphs - Smart Data
 
Using a Canary Microservice to Validate the Software Delivery Pipeline
Using a Canary Microservice to Validate the Software Delivery PipelineUsing a Canary Microservice to Validate the Software Delivery Pipeline
Using a Canary Microservice to Validate the Software Delivery Pipeline
 
Turnkey Riak KV Cluster
Turnkey Riak KV ClusterTurnkey Riak KV Cluster
Turnkey Riak KV Cluster
 
Demystifying Security Analytics: Data, Methods, Use Cases
Demystifying Security Analytics: Data, Methods, Use CasesDemystifying Security Analytics: Data, Methods, Use Cases
Demystifying Security Analytics: Data, Methods, Use Cases
 
Micro Services - Small is Beautiful
Micro Services - Small is BeautifulMicro Services - Small is Beautiful
Micro Services - Small is Beautiful
 
Tubular Labs - Using Elastic to Search Over 2.5B Videos
Tubular Labs - Using Elastic to Search Over 2.5B VideosTubular Labs - Using Elastic to Search Over 2.5B Videos
Tubular Labs - Using Elastic to Search Over 2.5B Videos
 
Heterogenous Persistence
Heterogenous PersistenceHeterogenous Persistence
Heterogenous Persistence
 
Combining sentences with the words although and despite
Combining sentences with the words although and despiteCombining sentences with the words although and despite
Combining sentences with the words although and despite
 
Bsides Delhi Security Automation for Red and Blue Teams
Bsides Delhi Security Automation for Red and Blue TeamsBsides Delhi Security Automation for Red and Blue Teams
Bsides Delhi Security Automation for Red and Blue Teams
 
Docker in Production, Look No Hands! by Scott Coulton
Docker in Production, Look No Hands! by Scott CoultonDocker in Production, Look No Hands! by Scott Coulton
Docker in Production, Look No Hands! by Scott Coulton
 
Deploying services: automation with docker and ansible
Deploying services: automation with docker and ansibleDeploying services: automation with docker and ansible
Deploying services: automation with docker and ansible
 
Adaptive Content Show & Tell - Austin Content
Adaptive Content Show & Tell - Austin ContentAdaptive Content Show & Tell - Austin Content
Adaptive Content Show & Tell - Austin Content
 
Build a Configurable Enterprise SaaS App in Only 9 Months
Build a Configurable Enterprise SaaS App in Only 9 MonthsBuild a Configurable Enterprise SaaS App in Only 9 Months
Build a Configurable Enterprise SaaS App in Only 9 Months
 
Resume
ResumeResume
Resume
 
SocCnx11 - All you need to know about orient me
SocCnx11 - All you need to know about orient meSocCnx11 - All you need to know about orient me
SocCnx11 - All you need to know about orient me
 
Secure Yourself, Practice what we preach - BSides Austin 2015
Secure Yourself, Practice what we preach - BSides Austin 2015Secure Yourself, Practice what we preach - BSides Austin 2015
Secure Yourself, Practice what we preach - BSides Austin 2015
 
"Mini Texts"
"Mini Texts" "Mini Texts"
"Mini Texts"
 
Accelerated Leadership
Accelerated LeadershipAccelerated Leadership
Accelerated Leadership
 

Similar to Splunk Dynamic lookup

Workshop: Big Data Visualization for Security
Workshop: Big Data Visualization for SecurityWorkshop: Big Data Visualization for Security
Workshop: Big Data Visualization for SecurityRaffael Marty
 
SplunkLive! Getting Started with Splunk Enterprise
SplunkLive! Getting Started with Splunk EnterpriseSplunkLive! Getting Started with Splunk Enterprise
SplunkLive! Getting Started with Splunk EnterpriseSplunk
 
Getting Started with Splunk Break out Session
Getting Started with Splunk Break out SessionGetting Started with Splunk Break out Session
Getting Started with Splunk Break out SessionGeorg Knon
 
Workshop splunk 6.5-saint-louis-mo
Workshop splunk 6.5-saint-louis-moWorkshop splunk 6.5-saint-louis-mo
Workshop splunk 6.5-saint-louis-moMohamad Hassan
 
SplunkLive! Zürich 2014 Beginner Workshop: Getting started with Splunk
SplunkLive! Zürich 2014 Beginner Workshop: Getting started with SplunkSplunkLive! Zürich 2014 Beginner Workshop: Getting started with Splunk
SplunkLive! Zürich 2014 Beginner Workshop: Getting started with SplunkGeorg Knon
 
Datacamp @ Transparency Camp 2010
Datacamp @ Transparency Camp 2010Datacamp @ Transparency Camp 2010
Datacamp @ Transparency Camp 2010Knowerce
 
Hadoop summit 2010, HONU
Hadoop summit 2010, HONUHadoop summit 2010, HONU
Hadoop summit 2010, HONUJerome Boulon
 
Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Apache Spark Data Source V2 with Wenchen Fan and Gengliang WangApache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Apache Spark Data Source V2 with Wenchen Fan and Gengliang WangDatabricks
 
MongoDB Days Germany: Data Processing with MongoDB
MongoDB Days Germany: Data Processing with MongoDBMongoDB Days Germany: Data Processing with MongoDB
MongoDB Days Germany: Data Processing with MongoDBMongoDB
 
Streaming data for real time analysis
Streaming data for real time analysisStreaming data for real time analysis
Streaming data for real time analysisAmazon Web Services
 
Berlin Buzz Words - Apache Drill by Ted Dunning & Michael Hausenblas
Berlin Buzz Words - Apache Drill by Ted Dunning & Michael HausenblasBerlin Buzz Words - Apache Drill by Ted Dunning & Michael Hausenblas
Berlin Buzz Words - Apache Drill by Ted Dunning & Michael HausenblasMapR Technologies
 
Machine Data 101 Hands-on
Machine Data 101 Hands-onMachine Data 101 Hands-on
Machine Data 101 Hands-onSplunk
 
How to use Parquet as a Sasis for ETL and Analytics
How to use Parquet as a Sasis for ETL and AnalyticsHow to use Parquet as a Sasis for ETL and Analytics
How to use Parquet as a Sasis for ETL and AnalyticsDataWorks Summit
 
Realtime Analytics on AWS
Realtime Analytics on AWSRealtime Analytics on AWS
Realtime Analytics on AWSSungmin Kim
 
Apache Big Data EU 2016: Building Streaming Applications with Apache Apex
Apache Big Data EU 2016: Building Streaming Applications with Apache ApexApache Big Data EU 2016: Building Streaming Applications with Apache Apex
Apache Big Data EU 2016: Building Streaming Applications with Apache ApexApache Apex
 
Getting started with Splunk - Break out Session
Getting started with Splunk - Break out SessionGetting started with Splunk - Break out Session
Getting started with Splunk - Break out SessionGeorg Knon
 
Getting started with Splunk
Getting started with SplunkGetting started with Splunk
Getting started with SplunkSplunk
 
Pivotal Real Time Data Stream Analytics
Pivotal Real Time Data Stream AnalyticsPivotal Real Time Data Stream Analytics
Pivotal Real Time Data Stream Analyticskgshukla
 

Similar to Splunk Dynamic lookup (20)

Workshop: Big Data Visualization for Security
Workshop: Big Data Visualization for SecurityWorkshop: Big Data Visualization for Security
Workshop: Big Data Visualization for Security
 
SplunkLive! Getting Started with Splunk Enterprise
SplunkLive! Getting Started with Splunk EnterpriseSplunkLive! Getting Started with Splunk Enterprise
SplunkLive! Getting Started with Splunk Enterprise
 
Getting Started with Splunk Break out Session
Getting Started with Splunk Break out SessionGetting Started with Splunk Break out Session
Getting Started with Splunk Break out Session
 
Workshop splunk 6.5-saint-louis-mo
Workshop splunk 6.5-saint-louis-moWorkshop splunk 6.5-saint-louis-mo
Workshop splunk 6.5-saint-louis-mo
 
SplunkLive! Zürich 2014 Beginner Workshop: Getting started with Splunk
SplunkLive! Zürich 2014 Beginner Workshop: Getting started with SplunkSplunkLive! Zürich 2014 Beginner Workshop: Getting started with Splunk
SplunkLive! Zürich 2014 Beginner Workshop: Getting started with Splunk
 
Datacamp @ Transparency Camp 2010
Datacamp @ Transparency Camp 2010Datacamp @ Transparency Camp 2010
Datacamp @ Transparency Camp 2010
 
Hadoop summit 2010, HONU
Hadoop summit 2010, HONUHadoop summit 2010, HONU
Hadoop summit 2010, HONU
 
20170126 big data processing
20170126 big data processing20170126 big data processing
20170126 big data processing
 
Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Apache Spark Data Source V2 with Wenchen Fan and Gengliang WangApache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang
 
Powering up on PowerShell - BSides Greenville 2019
Powering up on PowerShell  - BSides Greenville 2019Powering up on PowerShell  - BSides Greenville 2019
Powering up on PowerShell - BSides Greenville 2019
 
MongoDB Days Germany: Data Processing with MongoDB
MongoDB Days Germany: Data Processing with MongoDBMongoDB Days Germany: Data Processing with MongoDB
MongoDB Days Germany: Data Processing with MongoDB
 
Streaming data for real time analysis
Streaming data for real time analysisStreaming data for real time analysis
Streaming data for real time analysis
 
Berlin Buzz Words - Apache Drill by Ted Dunning & Michael Hausenblas
Berlin Buzz Words - Apache Drill by Ted Dunning & Michael HausenblasBerlin Buzz Words - Apache Drill by Ted Dunning & Michael Hausenblas
Berlin Buzz Words - Apache Drill by Ted Dunning & Michael Hausenblas
 
Machine Data 101 Hands-on
Machine Data 101 Hands-onMachine Data 101 Hands-on
Machine Data 101 Hands-on
 
How to use Parquet as a Sasis for ETL and Analytics
How to use Parquet as a Sasis for ETL and AnalyticsHow to use Parquet as a Sasis for ETL and Analytics
How to use Parquet as a Sasis for ETL and Analytics
 
Realtime Analytics on AWS
Realtime Analytics on AWSRealtime Analytics on AWS
Realtime Analytics on AWS
 
Apache Big Data EU 2016: Building Streaming Applications with Apache Apex
Apache Big Data EU 2016: Building Streaming Applications with Apache ApexApache Big Data EU 2016: Building Streaming Applications with Apache Apex
Apache Big Data EU 2016: Building Streaming Applications with Apache Apex
 
Getting started with Splunk - Break out Session
Getting started with Splunk - Break out SessionGetting started with Splunk - Break out Session
Getting started with Splunk - Break out Session
 
Getting started with Splunk
Getting started with SplunkGetting started with Splunk
Getting started with Splunk
 
Pivotal Real Time Data Stream Analytics
Pivotal Real Time Data Stream AnalyticsPivotal Real Time Data Stream Analytics
Pivotal Real Time Data Stream Analytics
 

More from Splunk

.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routineSplunk
 
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTVSplunk
 
.conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica).conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica)Splunk
 
.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank International.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank InternationalSplunk
 
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett .conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett Splunk
 
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär).conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)Splunk
 
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu....conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...Splunk
 
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever....conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...Splunk
 
.conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex).conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex)Splunk
 
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)Splunk
 
Splunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11ySplunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11ySplunk
 
Splunk x Freenet - .conf Go Köln
Splunk x Freenet - .conf Go KölnSplunk x Freenet - .conf Go Köln
Splunk x Freenet - .conf Go KölnSplunk
 
Splunk Security Session - .conf Go Köln
Splunk Security Session - .conf Go KölnSplunk Security Session - .conf Go Köln
Splunk Security Session - .conf Go KölnSplunk
 
Data foundations building success, at city scale – Imperial College London
 Data foundations building success, at city scale – Imperial College London Data foundations building success, at city scale – Imperial College London
Data foundations building success, at city scale – Imperial College LondonSplunk
 
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...Splunk
 
SOC, Amore Mio! | Security Webinar
SOC, Amore Mio! | Security WebinarSOC, Amore Mio! | Security Webinar
SOC, Amore Mio! | Security WebinarSplunk
 
.conf Go 2022 - Observability Session
.conf Go 2022 - Observability Session.conf Go 2022 - Observability Session
.conf Go 2022 - Observability SessionSplunk
 
.conf Go Zurich 2022 - Keynote
.conf Go Zurich 2022 - Keynote.conf Go Zurich 2022 - Keynote
.conf Go Zurich 2022 - KeynoteSplunk
 
.conf Go Zurich 2022 - Platform Session
.conf Go Zurich 2022 - Platform Session.conf Go Zurich 2022 - Platform Session
.conf Go Zurich 2022 - Platform SessionSplunk
 
.conf Go Zurich 2022 - Security Session
.conf Go Zurich 2022 - Security Session.conf Go Zurich 2022 - Security Session
.conf Go Zurich 2022 - Security SessionSplunk
 

More from Splunk (20)

.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine
 
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
 
.conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica).conf Go 2023 - Navegando la normativa SOX (Telefónica)
.conf Go 2023 - Navegando la normativa SOX (Telefónica)
 
.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank International.conf Go 2023 - Raiffeisen Bank International
.conf Go 2023 - Raiffeisen Bank International
 
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett .conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
.conf Go 2023 - På liv og død Om sikkerhetsarbeid i Norsk helsenett
 
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär).conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
.conf Go 2023 - Many roads lead to Rome - this was our journey (Julius Bär)
 
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu....conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...
 
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever....conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
.conf go 2023 - Cyber Resilienz – Herausforderungen und Ansatz für Energiever...
 
.conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex).conf go 2023 - De NOC a CSIRT (Cellnex)
.conf go 2023 - De NOC a CSIRT (Cellnex)
 
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
conf go 2023 - El camino hacia la ciberseguridad (ABANCA)
 
Splunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11ySplunk - BMW connects business and IT with data driven operations SRE and O11y
Splunk - BMW connects business and IT with data driven operations SRE and O11y
 
Splunk x Freenet - .conf Go Köln
Splunk x Freenet - .conf Go KölnSplunk x Freenet - .conf Go Köln
Splunk x Freenet - .conf Go Köln
 
Splunk Security Session - .conf Go Köln
Splunk Security Session - .conf Go KölnSplunk Security Session - .conf Go Köln
Splunk Security Session - .conf Go Köln
 
Data foundations building success, at city scale – Imperial College London
 Data foundations building success, at city scale – Imperial College London Data foundations building success, at city scale – Imperial College London
Data foundations building success, at city scale – Imperial College London
 
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...
 
SOC, Amore Mio! | Security Webinar
SOC, Amore Mio! | Security WebinarSOC, Amore Mio! | Security Webinar
SOC, Amore Mio! | Security Webinar
 
.conf Go 2022 - Observability Session
.conf Go 2022 - Observability Session.conf Go 2022 - Observability Session
.conf Go 2022 - Observability Session
 
.conf Go Zurich 2022 - Keynote
.conf Go Zurich 2022 - Keynote.conf Go Zurich 2022 - Keynote
.conf Go Zurich 2022 - Keynote
 
.conf Go Zurich 2022 - Platform Session
.conf Go Zurich 2022 - Platform Session.conf Go Zurich 2022 - Platform Session
.conf Go Zurich 2022 - Platform Session
 
.conf Go Zurich 2022 - Security Session
.conf Go Zurich 2022 - Security Session.conf Go Zurich 2022 - Security Session
.conf Go Zurich 2022 - Security Session
 

Recently uploaded

Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 

Recently uploaded (20)

Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 

Splunk Dynamic lookup

  • 2. Agenda Lookups in General Static Lookups Dynamic Lookups - Retrieve fields from a web site - Retrieve fields from a database - Retrieve fields from a persistent cache 2
  • 3. Enrich Your Events with Fields from External Sources 3
  • 4. Splunk: The Engine for Machine Data Customer Outside the Facing Data Datacenter Click-stream data Manufacturing, logistics Shopping cart data … Online transaction data CDRs & IPDRs Power consumption Logfiles Configs Messages Traps Metrics Scripts Changes Tickets RFID data Alerts GPS data Virtualization Windows Linux/Unix Applications Databases Networking & Cloud Registry Configurations Hypervisor Web logs Configurations Configurations Event logs syslog Guest OS, Apps Log4J, JMS, JMX Audit/query logs syslog File system File system Cloud .NET events Tables SNMP sysinternals ps, iostat, top Code and scripts Schemas netflow 4
  • 5. 5
  • 6. 6
  • 7. 7
  • 8. 8
  • 9. Interesting Things to Lookup • User’s Mailing Address • External Host Address • Error Code Descriptions • Database Query • Product Names • Web Service Call for Status • Stock Symbol (from CUSIP) • Geo Location 9
  • 10. Other Reasons For Lookup • Bypass static developer or vendor that does not enrich logs • Imaginative correlations • Example: A website URL with “Like” or “Dislike” count stored in external source • Make your data more interesting • Better to see textual descriptions than arcane codes 10
  • 11. Agenda Lookups in General Static Lookups Dynamic Lookups - Retrieve fields from a web site - Retrieve fields from a database - Retrieve fields from a persistent cache 11
  • 12. Static vs. Dynamic Lookup External Data comes from a CSV file Static Dynamic External Data comes from output of external script, which resembles a CSV file 12
  • 13. Static Lookup Review • Pick the input fields that will be used to get output fields • Create or locate a CSV file that has all the fields you need in the proper order • Tell Splunk via the Manager about your CSV file and your lookup • You can also define lookups manually via props.conf and transforms.conf • If you use automatic lookups, they will run every time the source, sourcetype or associated host stanza is used in a search • Non-automatic lookups run only when the lookup command is invoked in the search 13
  • 14. Example Static Lookup Conf Files props.conf [access_combined] lookup_http = http_status status OUTPUT status_description, status_type transforms.conf [http_status] filename = http_status.csv 14
  • 15. Permissions Define Lookups via Splunk Manager & set permissions there local.meta [lookups/http_status.csv] export = system [transforms/http_status] export = system 15
  • 17. Agenda Lookups in General Static Lookups Dynamic Lookups - Retrieve fields from a web site - Retrieve fields from a database - Retrieve fields from a persistent cache 17
  • 18. Dynamic Lookups • Write the script to simulate access to external source • Test the script with one set of inputs • Create the Splunk Version of the lookup script • Register the script with Splunk via Manager or conf files • Test the script explicitly before using automatic lookups 18
  • 19. Lookups vs Custom Command • Use dynamic lookups when returning fields given input fields • Standard use case for users who already are familiar with lookups • Use a custom command when doing MORE than a lookup • Not all use cases involve just returning fields • Decrypt event data • Translate event data from one format to another with new fields (e.g. FIX) 19
  • 20. Write/Test External Field Gathering Script Send: Input Fields External Data in Cloud Your Python Script Return: Output Fields 20
  • 21. Example Script to Test External Lookup # Given a host, find the corresponding IP address def mylookup(host): try: ipaddrlist = socket.gethostbyname_ex(host) return ipaddrlist except: return[] 21
  • 22. External Field Gathering Script with Splunk External Data in Cloud Your Python Script Return: Output Fields 22
  • 23. Script for Splunk Simulates Reading Input CSV hostname, ip a.b.c.com zorrosty.com seemanny.com 23
  • 24. Output of Script Returns Logically Complete CSV hostname, ip a.b.c.com, 1.2.3.4 zorrosty.com, 192.168.1.10 seemanny.com, 10.10.2.10 24
  • 25. transforms.conf for Dynamic Lookup [NameofLookup] external_cmd = <name>.py field1….fieldN external_type = python fields_list = field1, …, fieldN 25
  • 26. Example Dynamic Lookup conf files transforms.conf # Note – this is an explicit lookup [whoisLookup] external_cmd = whois_lookup.py ip whois external_type = python fields_list = ip, whois 26
  • 27. Dynamic Lookup Python Flow def lookup(input): Perform external lookup based on input. Return result main() Check standard input for CSV headers. Write headers to standard output. For each line in standard input (input fields): Gather input fields into a dictionary (key-value structure) ret = lookup(input fields) If ret: Send to standard output input values and return values from lookup 27
  • 28. Whois Lookup def main(): if len(sys.arv) != 3: print “Usage: python whois_lookup.py [ip field] [whois field]” sys.exit(0) ipf = sys.argv[1] whoisf = sys.argv[2] r = csv.reader(sys.stdin) w = none header = [ ] first = True… 28
  • 29. Whois Lookup (cont.) to Read CSV Header # First get read the “CSV Header” and output the field names for line in r: if first: header = line if whoisf not in header or ipf not in header: print “IP and whois fields must exist in CSV data” sys.exit(0) csv.write(sys.stdout).writerow(header) w = csv.DictWriter(sys.stdout, header) first = False continue… 29
  • 30. Whois Lookup (cont.) to Populate Input Fields # Read the result and populate the values for the input fields (ip address in our case) result = {} i=0 while i < len(header): if i < len(line): result[header[i]] = line[i] else: result[header[i]] = '' i += 1 30
  • 31. Whois Lookup (cont.) to Populate Input Fields # Perform the whois lookup if necessary if len(result[ipf]) and len(result[whoisf]): w.writerow(result) # Else call external website to get whois field from the ip address as the key elif len(result[ipf]): result[whoisf] = lookup(result[ipf]) if len(result[whoisf]): w.writerow(result) 31
  • 32. Whois Lookup Function LOCATION_URL=http://some.url.com?query= # Given an ip, return the whois response def lookup(ip): try: whois_ret = urllib.urlopen(LOCATION_URL + ip) lines = whois_ret.readlines() return lines except: return '' 32
  • 33. Database Lookup • Acquire proper modules to connect to the database • Connect and authenticate to database • Use a connection pool if possible • Have lookup function query the database • Return a list([]) of results 33
  • 34. Database Lookup vs. Database Sent To Index • Well, it depends… • Use a Lookup when: • Using needle in the haystack searches with a few users • Using form searches returning few results • Index the database table or view when: • Having LOTS of users and ad hoc reporting is needed • It’s OK to have “stale” data (N minutes) old for a dynamic database 34
  • 35. Example Database Lookup using MySQL # First connect to DB outside of the for loop conn = MySQLdb.connect(host = “localhost”, user = “name of user”, passwd = “password”, db = “Name of DB”) cursor = conn.cursor() 35
  • 36. Example Database Lookup (cont.) using MySQL import MySQLdb… # Given a city, find its country def lookup(city, cur): try: selString=“SELECT country FROM city_country where city=“ cur.execute(selString + “”” + city + “””) row = cur.fetechone() return row[0] except: return [] 36
  • 37. Lookup Using Key Value Persistent Cache • Download and install Redis • Download and install Redis Python module Redis is an open • Import Redis module in Python and populate source, advanced key- value store. key value DB • Import Redis module in lookup function given to Splunk to lookup a value given a key 37
  • 38. Redis Lookup ###CHANGE PATH According to your REDIS install ###### sys.path.append(“/Library/Python/2.6/…/redis-2.4.5-py.egg”) import redis … def main() … #Connect to redis – Change for your distribution pool = redis.ConnectionPool(host=„localhost‟,port=6379,db=0) redp = redis.Redis(connection_pool=pool) 38
  • 39. Redis Lookup (cont.) def lookup(redp, mykey): try: return redp.get(mykey) except: return “” 39
  • 40. Combine Persistent Cache with External Lookup • For data that is “relatively static” • First see if the data is in the persistent cache • If not, look it up in the external source such as a database or web service • If results come back, add results to the persistent cache and return results • For data that changes often, you will need to create your own cache retention policies 40
  • 41. Combining Redis with Whois Lookup def lookup(redp, ip): try: ret = redp.get(ip) if ret!=None and ret!='': return ret else: whois_ret = urllib.urlopen(LOCATION_URL + ip) lines = whois_ret.readlines() if lines!='': redp.set(ip, lines) return lines… except: 41
  • 42. Where do I get the add-ons from today? Splunkbase! Add-On Download Location Release http://splunk-base.splunk.com/apps/22381/whois- 4.x Whois add-on http://splunk- 4.x DBLookup base.splunk.com/apps/22394/example-lookup- using-a-database http://splunk-base.splunk.com/apps/27106/redis- 4.x Redis Lookup lookup http://splunk-base.splunk.com/apps/22282/geo- 4.x Geo IP Lookup (not location-lookup-script-powered-by-maxmind in these slides) 42
  • 43. Conclusion Lookups are a powerful way to enhance your search experience beyond indexing the data. 43

Editor's Notes

  1. Splunk is a data engine for your machine data. It gives you real-time visibility and intelligence into what’s happening across your IT infrastructure – whether it’s physical, virtual or in the cloud. Everybody now recognizes the value of this data, the problem up to now has been getting to it. At Splunk we applied the search engine paradigm to being able to rapidly harness any and all machine data wherever it originates. The “no predefined schema” design, means you can point Splunk at any of your data, regardless of format, source or location. There is no need to build custom parsers or connectors, there’s no traditional RDBMS, there’s no need to filter and forward.Here we see just a sample of the kinds of data Splunk can ‘eat’.Reminder – what’s the ‘big deal’ about machine data? It holds a categorical record of the following:User transactionsCustomer behaviorMachine behaviorSecurity threatsFraudulent activityYou can imagine that a single user transaction can span many systems and sources of this data, or a single service relies on many underlying systems. Splunk gives you one place to search, report on, analyze and visualize all this data.