# Folders structure
- experiments: In this folder you have to store all the recipes. At the moment the convention is to store each group of
  experiments in a separated folder. If you change this structure, you have to change the RELATIVE_BASE_DIR var the
  base_config file.
- idapythonrc: A configuration file for IDA which gives the hability to detect virtual environments and activate them.
  For more information visit: https://github.com/Kerrigan29a/idapython_virtualenv
- programs: This folder contains the programs used in every test. If you change this folder you have to change the 
  PROGRAMS_DIR var in the base_config file.
- scripts: This folder contains all the batch files to run automatically every experiment.
- tools: This folder contains all the python modules and tools used in this implmentation
    - colorful: Package to add colors to the terminal in Posix and Windows
    - instrumentator: This is program generates the instrumented files and clasifies the functions.
    - mongolog: This is the code in charge of storing the logs in Mongo DB. Look at the code of this module if you want
      to disable in a quick and dirty way all the logging system.
    - pyfare: This module is in charge of communicate with IDAPython and handle all the disassembler entities
      (functions, instructions, ...). This module can work in online mode (connected to the IDA disassembler) of in
      offline mode (connected to a File or to a Mongo DB) if all the entities has been loaded previously.
    - recipes: This module is the task orchestrator. All the recipes use this module to define the tasks, then runner
      inside this module distributes the work to the pool of workers.
    - uwkward: This is a utility module to help in some task related with unicode strings and encoding.
    - consolidate_dataset.py: This tool creates the final dataset.
    - dump_exe.py: This script is used to dump all the necessary entities from the IDA disassembler to the DB. From this
      moment all tools using Pyfare can work in offline mode.
    - extract_hits.py: This script extracts and generalizes the patterns. It's also in charge of generate the
      occurrences table
    - idaq.py: This script is an utility to wrap the idaq.exe process
    - log_viewer.py: Utility tool to handle and print the log from the framework
    - search_in_exe.py: Utility script to search instructions in the Mongo DB


# Framework installation
- Install Visual Studio. It will be used to compile binaries in the experiments.
- Install MongoDB 3.2 or greater
- Configure db_connection.json
- Install IDA Pro 6.1 or greater
- Install IDAPython
- Install idapythonrc with the provided script (install_idapythonrc.bat)
- Install Python 2 (or Anaconda)
- Install virtualenv (if necessary)
- Install packages from requirementes_extern.txt
- Install packages from requirementes_intern.txt


# Running an experiment
- Start the MongoDB Server
- Start the low_viewer (if you want to see what's happening)
- Run the experiment through the batch files in the scripts folder


# What's in the DB
- All the patterns and functions
- The final dataset
- Every log message


# Viewing logs
All the scripts store their logs in a separated database for each experiment.
To view the logs or search on them, use the log_viewer script


# Debugging with PyCharm
https://mipsme.blogspot.com/2015/03/ida-python.html