How Hard Should the Test Instances Be in Instance-Specific Macro Learning?

932 Words2 Pages

1 Introduction During the instance-specific macro learning experiments [1], we faced a problem in which there was no significant difference between the perfect model and the other models / macro sets. I thought initially that learning in general is not useful. But then I realized that this problem was caused partially by the way I collect data. The test examples were so easy to capture any significant difference in performance between the models. So, we need to make the test instances harder to solve in general. It is also possible to fall in the other extreme, which is making the problems too hard to the point where no model can solve most of the instance. Practically, for a test instance to be considered, I think I should fix a lower bound on the runtime of the empty set and an upper bound on the runtime of the perfect model in order to have a clear view of the differences between the models/macro sets. 2 Details We need to find hard-enough instances to test the macro performance. It is essential that the instances that I test are hard, because otherwise the differences in models performances might not be clear. This is easy to check: if the problem is really easily solvable on the empty set, I should not include it in the results. So I add a lower bound on the time of the empty set: T(i,m0) > MinTime But is this assumption correct? If we set this in advance before running the experiment, then maybe it is okay. My argument is: putting in mind that we want to measure the significance in the difference of performance between the models/macro sets, and given that the process switching time of current operating systems is non zero, we should make such an assumption. This is because there will be a small ove... ... middle of paper ... ...h instances, and it was hard to avoid generating such instances for the test. It is not possible to completely control the output of a random problem generator, and the mprime problems were either relatively easy or extremely hard. So, the only way I found to make things more fair in the comparison was to apply the upper bound on the perfect model as discussed above. This method was very effective in showing that the perfect model is superior compared to the other macros/model. Because in many of the instances where not every macro set timed out, the perfect model (or the imaginary best prediction model) solved the problem in a fraction of a second, where most of the other macros solved it in more than 50 minutes! This showed that removing the flawed information entries gave us a clearer picture and a more fair view of the power of instance-specific macros.

Open Document