Summary: Using Linux kernel ftrace to trace down kernel problems is essential in different ways, this post is to summarize how to build a LEDE / OpenWrt image for C2600 with ftrace enable (CONFIG_FUNCTION_TRACER, CONFIG_DYNAMIC_FTRACE, CONFIG_FUNCTION_GRAPH_TRACER, CONFIG_FUNCTION_PROFILER).
0x1 Prepare LEDE source code
Make sure you get LEDE source code from GitHub mirror or their git.
You can take a look at this post for detail: Build LEDE for TP-Link Archer C2600 from source (Chinese)
0x2 You will get failed if you just enable ftrace
You can use make menuconfig and go through Global build settings -> Kernel build options and check all option in Function tracer to enable.
And when you build with make -j8, you will get something like this:
There have nothing like openwrt-ipq806x-squashfs-sysupgrade.bin as usual. (Remember don’t use openwrt-ipq806x-squashfs-root.img with sysupgrade -f, this will brick your device and you will need to use tftp to debrick it)
0x3 Setting up config properly for C2600 with ftrace enable
Using make V=s to debug, we will find this output at the near end from tplink-safeloader:
|
1 2 |
/home/.../lede/staging_dir/host/bin/tplink–safeloader –B C2600 .... /tmp/openwrt–ipq806x–C2600–squashfs–sysupgrade.bin ... os–image partition too big (more than 2097152 bytes): Success |
Bingo, ftrace options make the firmware too large so that it can not let tplink-safeloader generate the file.
Adjust some config and rebuild again, then we can get the sysupgrade.bin file.
Following options are what I used:
- CONFIG_KERNEL_KALLSYMS (so that you can trace with driver, mac80211 …etc)
- KERNEL_FTRACE
- KERNEL_FUNCTION_TRACER
- KERNEL_FUNCTION_GRAPH_TRACER
- KERNEL_DYNAMIC_FTRACE
- KERNEL_FUNCTION_PROFILER
Following config file is what I used:
0x4 Using ftrace on C2600 with LEDE
To record mac80211 and ath10k events
|
1 2 |
$ trace–cmd record –e mac80211 –e ath10k $ trace–cmd report |
To record specific function and draw out the function graph
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
$ trace–cmd record –a –p function_graph –g ath10k_mac_op_wake_tx_queue $ trace–cmd report trace–cmd: Invalid argument [ras:mc_event] function mc_event_error_type not defined trace–cmd: Invalid argument [libata:ata_qc_issue] function libata_trace_parse_subcmd not defined [libata:ata_qc_complete_internal] function libata_trace_parse_qc_flags not defined [libata:ata_qc_complete_failed] function libata_trace_parse_qc_flags not defined [libata:ata_qc_complete_done] function libata_trace_parse_qc_flags not defined [libata:ata_eh_link_autopsy_qc] function libata_trace_parse_qc_flags not defined [libata:ata_eh_link_autopsy] function libata_trace_parse_eh_action not defined trace–cmd: Invalid argument [dwc3:dwc3_prepare_trb] bad op token { [dwc3:dwc3_gadget_generic_cmd] function dwc3_gadget_generic_cmd_string not defined [dwc3:dwc3_gadget_ep_cmd] function dwc3_gadget_ep_cmd_string not defined [dwc3:dwc3_event] function dwc3_decode_event not defined [dwc3:dwc3_complete_trb] bad op token { cpus=2 <idle>–0 [001] 394.985361: funcgraph_entry: | ath10k_mac_op_wake_tx_queue() { <idle>–0 [001] 394.985479: funcgraph_entry: 2.720 us | _raw_spin_lock_bh(); <idle>–0 [001] 394.985498: funcgraph_entry: | ath10k_mac_tx_push_txq() { <idle>–0 [001] 394.985505: funcgraph_entry: 1.600 us | _raw_spin_lock_bh(); <idle>–0 [001] 394.985517: funcgraph_entry: 1.920 us | ath10k_htt_tx_inc_pending(); <idle>–0 [001] 394.985529: funcgraph_entry: | _raw_spin_unlock_bh() { <idle>–0 [001] 394.985535: funcgraph_entry: 1.440 us | __local_bh_enable_ip(); <idle>–0 [001] 394.985546: funcgraph_exit: + 12.320 us | } <idle>–0 [001] 394.985552: funcgraph_entry: | ieee80211_tx_dequeue() { |
More references:
- ath10k architecture
- ath10k tracing
- Debugging the kernel using Ftrace – part 1
- Debugging the kernel using Ftrace – part 2
- Secrets of the Ftrace function tracer
- Advanced Features of Ftrace
0x5 Special Thanks
Many thanks to #lede-dev @hanetzer，hanetzer kindly help me in the first timing when I post the problem on the channel, and give me many useful advice on LEDE and images. Without hanetzer I will not able to enable this. Thanks.
Leave a Reply