Results 1 to 6 of 6
 0 Attachment(s)    

Thread: How to have a 99% modelling quality

  1. #1
    Administrator funyoo's Avatar
    Join Date
    Sep 2008

    Default How to have a 99% modelling quality

    Here is a guide supposed to allow you to reach the modelling quality of 99%.

    For the moment, it is only a myth. Up to now, we have never seen a modelling quality of 99%.
    Why backtest with 99% modelling quality?

    Because it is far more accurate than the widely employed 90% modelling quality approach that uses 1M data and the MT4 strategy tester fractal interpolation option.
    It is unfortunately quite easy to create EAs that inadvertently exploit the fractal interpolation algorithm to generate grossly unrealistic profits in backtests. They are typically scalpers that close large numbers of trades with less than 20 pips of profit or loss. Most perform far worse, or are not profitable at all, in live trading.

    Many commercial EAs are sold on the basis of these 90% modelling backtests, and the forex forums are littered with disappointed comments from people who did not see the same results when forward testing or trading live.

    Because most MT4 users know that something is seriously wrong with 90% modelling quality, a huge amount of energy goes into forward testing EAs that were never destined to be profitable. Forward testing is usually still a necessary final step before committing an EA to live trading, but it is my hope that, by describing this method, traders will be able to focus more productively on truly profitable strategies.

    How to get 99% modelling quality in the MT4 strategy tester

    Obtain tick data

    Logging tick data

    The best quality tick data comes from your broker’s live server, but unfortunately most brokers don’t supply it. The reason you need this is that each broker has different data feeds and applies different tick filtering.

    This logging EA will record ticks against whatever currency pair you attach it to. The timeframe doesn’t matter.

    It stores log files in experts\files\AAABBB_yyyymmdd_ticks.csv, where AAABBB is the currency pair that you are logging and yyyymmdd is the date you started.
    Over time you will get a number of tick log files with different dates. They can be merged into one file using a Windows XP DOS command like this:
    copy AAABBB_yyyymmdd_ticks.csv + AAABBB_yyyymmdd_ticks.csv + AAABBB_yyyymmdd_ticks.csv AAABBB_ticks.csv
    Note that if you're serious about logging ticks (and more importantly, live trading) then you need to schedule windows updates for only on the weekend.

    You want to backtest using tick data now, but you have no data?

    A less accurate alternative is to obtain tick data from a supplier, but the advantage is that an existing large historical record exists.

    Here is a free source of tick data The quality may be suspect.

    This script will convert the Gain files into FXT format

    Drop it on the chart timeframe that you want the FXT file to be generated in, then copy the AAABBBnn_0.fxt file from experts\files to tester\history

    Here you can pay for tick data: Presumably this would be good quality but I haven’t tried them.

    Converting tick data into the strategy tester FXT history file format

    Store this Metaquotes converter (simple_csv2fxt.mq4) in experts\scripts and compile: It requires this header file (FXTheader.mqh) to be stored in experts\include:

    You will get a “Function ReadAndCheckHeader is not referenced” warning, which can be ignored because the function exists in the header file but is not used.

    To run the script, first you open the chart for the timeframe and currency pair that you want to test. You then drop the script onto the chart and enter a filename of the logged ticks. This will look like AAABBB_yyyymmdd_ticks.csv. The script produce a filename like AAABBB30_0.fxt in experts\files, The 30 is for 30M timeframe, and the 0 is for tick level. This file then needs to be moved to tester\history. Copy over the existing file if it’s there.

    Running the backtest on real ticks

    Select a currency pair and timeframe for which you have generated an FXTfile, uncheck “recalculate” and observe the results. Likewise for optimisation. The report should show a modelling accuracy of 99%.

  2. #2
    Administrator funyoo's Avatar
    Join Date
    Sep 2008


    The second part.

    Which EAs should be backtested using this method
    • Any EA which frequently closes trades with less than 20 pips of profit or loss is likely to show a very different result using the 90% modelling quality that comes from fractal interpolation.
    • Any EA which could enter and exit within the same 1M candle.
    An explanation of why this is so is further down in this document.

    Why is it only 99% modelling accuracy?

    The figure of 99% is a notional one, but even a backtest done with live data from your own broker is certainly not 100% accurate. Here are some factors that affect the accuracy of backtests:
    • The EA start() function only receives a tick if it has completed processing the last one. It will therefore skip over ticks where the broker’s server is responding slowly, or the internet connection or PC is too slow.
    • Backtesting assumes perfect fills at the Bid or Ask, and this is not always the case in live trading.
    How does the MT4 strategy tester perform backtests?

    The mechanism

    A backtest should as much as possible try to simulate what would have happened if you traded live in the past. When you trade live, the start() function in your EA is executed every time a tick arrives, regardless of the timeframe to which your EA is attached. So the most accurate backtest will read historical ticks from a file and submit them one by one to the start() function.

    The strategy tester has three methods of repeatedly supplying tick data to the start() function, each with different levels of backtesting accuracy. The main reason for this is that the most accurate is also the slowest.

    We will focus on the “fractal interpolation of ticks” method, which is the slowest and most accurate model. A procedure for backtesting from 1M data using this method is here:

    Here’s what happens when you perform a 90% modelling backtest as described above, which occurs when you check “recalculate” in the strategy tester window and have sufficient data in the 1M timeframe and the timeframe required by your EA.
    • The MT4 strategy tester reads the 1M data file stored in the history folder
    • It applies an algorithm called fractal interpolation to generate ticks within each 1M bar. If we are testing EURUSD on a 30M timeframe it will store these in tester\history\EURUSD30_0.fxt, where 30 is the timeframe, and 0 means fractal interpolation.
    • It then submits ticks from this file to the start() function in your EA
    Flaws in the 90% modelling fractal interpolation approach
    • Anything can happen in a 1M candle
    The fractal interpolation algorithm attempts to guess what happened in a 1M candle by simply using the 1M Open/High/Low/Close data, and clearly this may not be anything like what really happened. The figure below gives just one situation

    If your EA has entry and exit levels at the point shown, then you will get two trades on live data, but only one in the simulation. So if your EA has any possibility of entering and exiting a trade within 1 minute, then fractal interpolation will give highly inaccurate results. And remember that 1M candles frequently have a range of 50 pips during news announcements.

    Fractal interpolation supplies ticks with larger jumps than reality

    The following two graphs show the difference in distribution of tick jumps between live and fractal interpolation .7.2006%20to%207.8.2006.GIF 006.GIF

    About 90% of tick movements on a 1 week record of live USDJPY data (see above) were 1 pip. This means that the takeprofit level in an EA would have been the exact value used in the order exit 90% of the time.

    In contrast, the fractal interpolation of ticks from 1M data delivered only 68% of ticks with a 1 pip movement. This means that the price could have exceeded the takeprofit value 32% of the time, delivering a higher profit. In a scalper that may take hundreds or thousands of profits of only a few pips, this grossly inflates the estimates of total profit.

    Bypassing fractal interpolation to supply historical ticks for 99% modelling

    The MT4 strategy tester reads tick data from files in the tester\history folder of the format AAABBBnn_0.fxt, where AAABBB is the currency pair, nn is the timeframe, and 0 means that the file was created using fractal interpolation. This 0 = fractal interpolation code is only correct if you generated it by checking “recalculate” in the strategy tester window. If you created it from your own tick data then it contains real tick data, and you should backtest using “recalculate” unchecked.

    Backtesting with 99 percent modelling V1.doc Paul Hampton-Smith, October 2006

  3. #3
    Junior Member
    Join Date
    Feb 2009

    Unhappy only original broker datas are worth to test ea`s

    hello funyoo

    i`m new here , first thanks for your work in this forum , it`s great and very good informations please can everyone help with a new downloadlink for the logger file TickLoggerForFXT.mq4 ??
    your postet downloadlink does not exist .......thanks a lot ...

    if anyone have original datas of different brokers for backtesting EA`s please help me with downloadlink or have anyone logged original tickdatas from different brokers ?? ....
    i have testet with metaquote metatrader download datas and i think it is none worth the time of testing ......i have testet some EA`s with metaquote datas with very good results and after testing example clearing, ibfx demo account`s it`s bad results ......
    it`s important to test ibfx account with original ibfx datas, clearing with clearing datas and so on ....

    thanks a lot for help and every suggestion

    nice trading and thanks


  4. #4
    Junior Member
    Join Date
    Jan 2009

  5. #5

    Default Link not working

    Thanks for the hard work on the tick data gathering script. Very kind of you to offer this work, but like another member here, it would bee great if you could reset the link, as it does not work at the moment. Hope you can help. Thanks.

  6. #6
    Junior Member
    Join Date
    May 2011


    hello, i m new here, so far I know that dukascopy offers tick level data and there is a converter to convert it into M1 data, see Tick data - 99% modeling quality with Metatrader 4 | Birt's EA review ... since MT4 can only be imported with M1 data, may I know have anybody backtested scalper like the channel scalper too see its 99% (tick level) performance? do you think broker will fake M1 data for your backtesting? Please advise... thank you very much

Similar Threads

  1. Replies: 1
    Last Post: 08-20-2014, 14:37
  2. 99 % Modelling HELP
    By FX-BoT-FX in forum Trading discussion
    Replies: 0
    Last Post: 11-19-2011, 23:54
  3. How to have a 90% modelling quality
    By funyoo in forum Trading discussion
    Replies: 0
    Last Post: 11-17-2008, 07:58

Tags for this Thread

100, backtest, backtesting, broker, brokers, candle, channel, color, copy, date, demo, eas, eurusd, expert, experts, forex, free, guide, high, historical, how to, level, live, main, metatrader, mq4, mql, mql4, mt4, news, offer, pips, profit, profitable, report, scalper, scripts, server, simple, source, strategies, strategy, test, thank you, time, trading

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts