This presentation shows that code coverage guided fuzzing is possible in the context of network daemon fuzzing. Some fuzzers are blackbox while others are protocol aware. Even ones which are made protocol aware, fuzzer writers typically model the protocol specification and implement packet awareness logic in the fuzzer. Unfortunately, just because the fuzzer is protocol aware, it does not guarantee that sufficient code paths have been reached. The presentation deals with specific scenarios where the target protocol is completely unknown (proprietary) and no source code or protocol specs are accessible. The tool developed builds a feedback loop between the client and the server components using the concept of "gate functions". A gate function triggers monitoring. The pintool component tracks the binary code coverage for all the functions untill it reaches an exit gate. By instrumenting such gated functions, the tool is able to measure code coverage during packet processing.

1. Beyond the Tip of the IceBerg
Fuzzing Binary Protocol for Deeper Code Coverage
Mrityunjay Gautam .Alex Moneger

2. Who we are?
• Security Engineers at Citrix Systems, Inc.
• Interest in low level topics (crypto, fuzzing, exploit dev)
Disclaimer
The views expressed herein are personal and stated in our
individual capacity and in no way a statement or position of
Citrix Systems, Inc.

3. Agenda
1. State of Fuzzers and Fuzzing Technology
2. Code Coverage based Fuzzers – AFL, et al
3. Binary Code Tracing: Gate Function
4. Applications to fuzzing – Feedback Loop
5. PoC Demo – Toy Example
6. Heuristic based Protocol Analysis

4. FUZZING AS WE KNEW IT

5. Fuzzing: Myths Vs Reality
• Myth: “fuzzing is easy”:
– flip some bits
– Collect bugs
• Reality: “fuzzing is complex”:
– Identifying target functions & writing wrapper code
– Building and minimizing a corpus
– Minimizing test-cases
– Instrumentation:
• Is input X better then Y?
• Did my application crash on input X or Y?

6. File format fuzzing
• Lots of focus on parsers:
– American Fuzzy Lop
– Honggfuzz
• Handling network code with them is tricky

7. Network fuzzing
• Still stuck modeling protocols
• Still slow
• Still requires some sort of agent to detect crashes
• We’re still blind fuzzing
• Yet, network stack is a target of choice
• We need more balance

8. Historically…
• 2 approaches:
– Mutate data forever (randomly, byte flip, …)
– Model data, mutate fields separately (Spike, Peach, Codenomicon,
…): Anyone written a complex Peach pit?
• Run for some iterations or until all states are modeled
• Hope for the best
• Claim that you have covered 10n iterations and feel good about
it 

9. FUZZING TODAY

10. Today
Genetic algorithms => retain only best input for further ‘mutation’
1. Mutate best input
2. Send to target
3. Measure fitness() based on Heuristics
4. Discard or prioritize input, back to 1.
We know how inputs affect target!

11. Fitness Heuristic: Code coverage
• Code coverage is the most used metric
• Tells you if an input has triggered new code paths
• All tools try to measure code coverage one way or another
• Can be achieved :
– Binary instrumentation (PIN, DynamoRIO)
– Static rewriting (Dyninst)
– Kernel probing (perf)
– HW (intel BTS => branch trace store)

12. How does it work
• Model control flow using basic blocks:
• Discard unconditional edges (JMPs)
• Retain edge count
• Provides an unordered code coverage map
• Code coverage are sets which can be compared:
– 0x08040302 => 08040301 : 1
– 0x0804030b => 08040404 : 5
0x08040302 0x08040301

13. Thanks AFL
• AFL revolutionized fuzzing
• “Batteries included” fuzzer
• Perfect balance between:
– Using build systems
– Speed
– Functionality
• Caveat: compares traces across runs:
– Target has to exit
– Has to get data off stdin

14. PROBLEM???

15. Limitations
• If you have source code, get AFL to work on packets
• Write wrappers, handle state, exit, … not pretty, but kind of
works
• Tight coupling may force to stub out function
– using LD_PRELOAD (see preeny)
– using linker -Wl,-wrap

16. Network daemons
• A solution could be to change the model a bit
• Keep successful AFL concepts:
– Code coverage
– Genetic algorithm
• But avoid restarting the target
• This breaks the deterministic nature of AFL

17. Requirements
• Improve traditional fuzzers:
– Get rid of the “try single input then check” cycle
• By borrowing from feedback driven fuzzers:
– Code coverage
– Genetic algorithm
• Do this during runtime
• Without re-spawning the target between inputs

18. OUR APPROACH

19. Observations
• Network daemon operations:
1. Do startup stuff,
2. Wait for connection
1. Connection establishment
2. Wait for input (read)
3. Process input packet
4. Send something based on input (write)
5. Loop from 2.2 till connection closes
3. Close (close) and go to 2.
• What code coverage do we care
about?
• Trace code between first read (2.2)
and last write (2.4)?
Startup
Read
Write
Close
Parse

20. Gate functions
• Here read()/write() can be considered gates
• When you enter a gate, trace
• When you exit a gate, stop trace
• Transfer code coverage to decision maker

21. Generalized approach
• Trigger code coverage collection at runtime
• Based on defined “gate” syscalls, say X and Y
• When syscall X is triggered, start recording edge transitions
• When syscall Y is triggered, stop recording
• Dump trace
• Repeat

22. 1000 feet view
• Track only network file descriptors
• Ignore I/O FDs
• Generate a hitmap at runtime through “gate” syscalls
• Dump it to fuzzer for analysis
• Fuzzer elects best input

23. Filtering file descriptors
• Accept() syscall returns FD
• Track FDs returned
• Checked if they’re passed in to:
– Read
– Write
• Stop tracking on close()
Accept 6,7,86
Read(6)
Write(6)
Read(9)
Write(9)

24. Aggregatemap
Coverage map
• Coverage maps are per
read/write gate
• You get several maps for one
connection
• Allows fuzzing a specific state
• Can also aggregate code
coverage between gate functions
Accept
Read(6)
Write(6)
Read(6)
Write(6)
Read(6)
Close(6)
Map 1
Map 2
Map 3

25. Ugly diagram
Accept
=> 6
6, 7, …, fd
Read(6)
Write(6)
Close(6)
Heat Map
Network FD list
Do stuff

26. UDP
• Exact same thing, but track:
– Recvfrom/recvmsg
– Sendto/sendmsg
• Generalization is possible to any syscall sequence
• Could use similar grammar to seccomp BPF

27. Netcov
• “Simple” pintool: https://github.com/alexmgr/netcov
• Generate code coverage maps at runtime
• Write them to a pipe
• Reverse of fuzzing talks, here fuzzing is up to you ;)
• Sidekick: netcallgraph:
– Generates runtime callgraph
• A dummy fuzzing example:
https://github.com/alexmgr/netcov-client

28. It’s a PoC…
• Limitations:
– Read hangs
– Select/poll
– No crash detection
– No ASAN to catch memory errors
– Hit map format is text based
• Works well:
– Multithreaded daemons
– Heatmap is per FD=> allows concurrent fuzzing
– Mutation independent
– Source code independent
• It’s a demo, not a tool

29. Netcov flow
Netcov
Daemon
Client (Fuzzer,
…)
Coverage
Protocol

30. Demo
• Demo daemon, magic packet: “ABC1234567890i”:
if (read(conn_desc, buff, sizeof(buff) - 1) > 0) {
printf("Received %sn", buff);
if (buff[0] == 'A') {
printf("Took first branchn");
if (buff[1] == 'B') {
printf("Took second branchn");
if (buff[2] == 'C') {
printf("Took third branchn");
if (strncmp(buff + 3, "1234567890", 10) == 0) {
printf("Good job!n");
char *num = buff + 13;
printf("Got num: %dn", atoi(num));
int i = 0;
for (i = 0; i < atoi(num); i++) {
printf("%d..", i);
}
write(conn_desc, "Good job!", 10);

31. Example netcallgraph

32. Fuzzing demo
• Start with an input value
• Byteflip it
• Measure coverage
1. If coverage increases, keep as best input
2. Mutate
3. Repeat 1.

33. REAL WORLD EXAMPLE – RDP PROTOCOL

34. RDP – Remote Desktop Protocol
• TCP Protocol on port 3389
• Originally on Windows variants
• Ported to most Unix Environments – XRDP
• Clients available on all Linux, Mac, Windows flavors

35. Weaponizing the ‘netcov’ PoC
Send Next
Mutated Packet XRDP Server
Netcov Binary Tracing
/tmp/netcovmap
Receive Binary Trace
between (recv, send)
Fitness function
(Unique Code Coverage)
Feedback on Packet
Quality
Load RDP
Wireshark Trace
Identify Packet to
Play With
Mutation Strategy
– Based on
Feedback
Process
Feedback
Result
Generation
Synchronization
Problem

36. XRDP Packet Analysis Results
Restricting the trace to libxrdp ONLY
Base Pkt:
0300002621e00000000000436f6f6b69653a206d737473686173683d0d0a0100080003000000
Baseline:
write:8=libxrdp.so.0+14816->libxrdp.so.0+14840:1;libxrdp.so.0+14840-
>libxrdp.so.0+14881:1;libxrdp.so.0+14881->libxrdp.so.0+47232:1;libxrdp.so.0+14904-
>libxrdp.so.0+14908:1;libxrdp.so.0+14908->libxrdp.so.0+14924:1;libxrdp.so.0+14924-
>libxrdp.so.0+14949:1;libxrdp.so.0+14949->libxrdp.so.0+14989:1;libxrdp.so.0+14989-
>libxrdp.so.0+15369:1;libxrdp.so.0+15348->libxrdp.so.0+15352:1;libxrdp.so.0+15352-
>libxrdp.so.0+14816:1;libxrdp.so.0+15369->libxrdp.so.0+15424:1;libxrdp.so.0+15424-
>libxrdp.so.0+15434:1;libxrdp.so.0+15434->libxrdp.so.0+47152:1;libxrdp.so.0+15446-
>libxrdp.so.0+15450:1;libxrdp.so.0+15450->libxrdp.so.0+47344:1;libxrdp.so.0+47152-
>libxrdp.so.0+47165:1;libxrdp.so.0+47165->libxrdp.so.0+15446:1;libxrdp.so.0+47232-
>libxrdp.so.0+47249:1;libxrdp.so.0+47249->libxrdp.so.0+47280:1;libxrdp.so.0+47280-
>libxrdp.so.0+14904:1;libxrdp.so.0+47280->libxrdp.so.0+15348:1;

37. Results
Packet 0: (To RDP
Server)
[(0, 0, 'CONTROL'),
(1, 1, 'DATA'),
(2, 3, 'MAGIC'),
(4, 4, 'DATA'),
(5, 5, 'CONTROL'),
(6, 37, 'DATA')]

38. Results
Packet 0: (To RDP
Server)
[(0, 0, 'CONTROL'),
(1, 1, 'DATA'),
(2, 3, 'MAGIC'),
(4, 4, 'DATA'),
(5, 5, 'CONTROL'),
(6, 37, 'DATA')]

39. Results
Packet 0: (To RDP
Server)
[(0, 0, 'CONTROL'),
(1, 1, 'DATA'),
(2, 3, 'MAGIC'),
(4, 4, 'DATA'),
(5, 5, 'CONTROL'),
(6, 37, 'DATA')]

40. XRDP Implementation Analysis
• Analysis of the 1st Packet:
– Byte (1) mutation leads to control flow change
– Bytes (3,4) are length of the packet. Verified before further
processing.
– Byte (5) is length of x224CRQ Header. Not verified before processing
or may lead to over-read.
– Byte (6) mutation leads to control flow change
– Bytes (7,38) is DATA. Fuzzable with different Control Flow bits.

41. Who in the room cannot write a fuzzer now ?

42. CONCLUSION

43. Conclusion
• Much to do in the world of network fuzzing
• Still stuck with:
– Dumb mutation fuzzers
– Model based fuzzers
– Slowness
• We present “just” a glimpse of what CAN be achieved

44. Thank You 