Nokogiri :: XML :: Reader - 큰 XML 파일 처리 및 아무런 관심이없는 노드 건너 뛰기

Question

파일 크기가 Nokogiri :: XML :: Reader를 사용하여 구문 분석하려고하는 아래와 같은 형식의 XML이 있습니다. 매우 거대한 (~ 1GB). 파일에는 아래 형식의 많은 packets이 있습니다.

각 packet에서 frame.time_epoch, s1ap.procedureCode을 수집해야합니다.

현재 다음 작업을 수행하고 있습니다.

data = [] 
file = `some_file.xml` 
reader = Nokogiri::XML::Reader(File.open(file)) 
reader.each do |node| 
    if (node.name == 'packet' && node.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT) 
     doc = Nokogiri::XML(node.outer_xml) 
     next if !doc.css("field[name='s1ap.procedureCode']") ## do nothing if the <packet> is not of s1ap type 
     epochTime = doc.css("field[name='frame.time_epoch']").first["show"].to_i 
     procedureCode = procedureCode_node = doc.css("field[name='s1ap.procedureCode']").first["show"].to_i 
     data << { epochTime: epochTime, procedureCode: procedureCode } 
    end 
end 

문제는

내가 직면하고있어 문제는 구문 분석이 정말 느린 것입니다. 한 가지 주목할 점은 독자가 <packet> </packet> 내의 모든 후속 라인을 스캔한다는 것입니다. 독자가 packet이라는 이름으로 다음 노드로 이동하게하는 방법이 있습니까? packet 내 각 행을 계속 진행하는 것입니다. 당신은 메모리에 모든 일을 당기고로 해석하지 않고 큰 문서를 처리 SAX 파서

http://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/XML/SAX

스트림을 사용해야하는 등 큰 문서에 대한

XML 형식

<?xml version="1.0" encoding="utf-8"?> 
<?xml-stylesheet type="text/xsl" href="pdml2html.xsl"?> 
<packet> 
    <proto name="geninfo" pos="0" showname="General information" size="126"> 
    <field name="num" pos="0" show="6" showname="Number" value="6" size="126"/> 
    </proto> 
    <proto name="frame" showname="Frame 6: 126 bytes on wire (1008 bits), 126 bytes captured (1008 bits) on interface 0" size="126" pos="0"> 
    <field name="frame.encap_type" showname="Encapsulation type: Ethernet (1)" size="0" pos="0" show="1"/> 
    <field name="frame.time_epoch" showname="Epoch Time: 1474267259.184197000 seconds" size="0" pos="0" show="1474267259.184197000"/> 
    </proto> 
    <proto name="eth" showname="Ethernet II, Src: JuniperN_e6:a6:cc (40:b4:f0:e6:a6:cc), Dst: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="14" pos="0"> 
    <field name="eth.dst" showname="Destination: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="6" pos="0" show="ac:16:2d:89:a5:91" value="ac162d89a591"> 
     <field name="eth.dst_resolved" showname="Destination (resolved): HewlettP_89:a5:91" hide="yes" size="6" pos="0" show="HewlettP_89:a5:91" value="ac162d89a591"/> 
    </field> 
    <field name="eth.type" showname="Type: IPv4 (0x0800)" size="2" pos="12" show="0x00000800" value="0800"/> 
    </proto> 
    <proto name="s1ap" showname="S1 Application Protocol" size="45" pos="78"> 
    <field name="per.choice_index" showname="Choice Index: 0" hide="yes" size="1" pos="78" show="0" value="00"/> 
    <field name="s1ap.S1AP_PDU" showname="S1AP-PDU: initiatingMessage (0)" size="45" pos="78" show="0" value="000b402900000300000005c007c03ae900080003403b53001a0012113743f99f9500075d010605f070c04070c1"> 
     <field name="s1ap.initiatingMessage_element" showname="initiatingMessage" size="45" pos="78" show="" value=""> 
     <field name="s1ap.procedureCode" showname="procedureCode: id-downlinkNASTransport (11)" size="1" pos="79" show="11" value="0b"/> 
     </field> 
    </field> 
    </proto> 
</packet> 
<packet> 
    <proto name="geninfo" pos="0" showname="General information" size="126"> 
    <field name="num" pos="0" show="6" showname="Number" value="6" size="126"/> 
    </proto> 
    <proto name="frame" showname="Frame 6: 126 bytes on wire (1008 bits), 126 bytes captured (1008 bits) on interface 0" size="126" pos="0"> 
    <field name="frame.encap_type" showname="Encapsulation type: Ethernet (1)" size="0" pos="0" show="1"/> 
    <field name="frame.time_epoch" showname="Epoch Time: 1474267260.184197000 seconds" size="0" pos="0" show="1474267259.184197000"/> 
    </proto> 
    <proto name="eth" showname="Ethernet II, Src: JuniperN_e6:a6:cc (40:b4:f0:e6:a6:cc), Dst: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="14" pos="0"> 
    <field name="eth.dst" showname="Destination: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="6" pos="0" show="ac:16:2d:89:a5:91" value="ac162d89a591"> 
     <field name="eth.dst_resolved" showname="Destination (resolved): HewlettP_89:a5:91" hide="yes" size="6" pos="0" show="HewlettP_89:a5:91" value="ac162d89a591"/> 
    </field> 
    <field name="eth.type" showname="Type: IPv4 (0x0800)" size="2" pos="12" show="0x00000800" value="0800"/> 
    </proto> 
    <proto name="s1ap" showname="Some other protocol" size="45" pos="78"> 
    <field name="per.choice_index" showname="Choice Index: 0" hide="yes" size="1" pos="78" show="0" value="00"/> 
    <field name="other.OTH_PDU" showname="S1AP-PDU: initiatingMessage (0)" size="45" pos="78" show="0" value="000b402900000300000005c007c03ae900080003403b53001a0012113743f99f9500075d010605f070c04070c1"> 
     <field name="other.initiatingMessage_element" showname="initiatingMessage" size="45" pos="78" show="" value=""> 
     <field name="other.procedureCode" showname="procedureCode: id-someTransport (99)" size="1" pos="79" show="11" value="0b"/> 
     </field> 
    </field> 
    </proto> 
</packet> 
<packet> 
    <proto name="geninfo" pos="0" showname="General information" size="126"> 
    <field name="num" pos="0" show="6" showname="Number" value="6" size="126"/> 
    </proto> 
    <proto name="frame" showname="Frame 6: 126 bytes on wire (1008 bits), 126 bytes captured (1008 bits) on interface 0" size="126" pos="0"> 
    <field name="frame.encap_type" showname="Encapsulation type: Ethernet (1)" size="0" pos="0" show="1"/> 
    <field name="frame.time_epoch" showname="Epoch Time: 1474267261.184197000 seconds" size="0" pos="0" show="1474267259.184197000"/> 
    </proto> 
    <proto name="eth" showname="Ethernet II, Src: JuniperN_e6:a6:cc (40:b4:f0:e6:a6:cc), Dst: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="14" pos="0"> 
    <field name="eth.dst" showname="Destination: HewlettP_89:a5:91 (ac:16:2d:89:a5:91)" size="6" pos="0" show="ac:16:2d:89:a5:91" value="ac162d89a591"> 
     <field name="eth.dst_resolved" showname="Destination (resolved): HewlettP_89:a5:91" hide="yes" size="6" pos="0" show="HewlettP_89:a5:91" value="ac162d89a591"/> 
    </field> 
    <field name="eth.type" showname="Type: IPv4 (0x0800)" size="2" pos="12" show="0x00000800" value="0800"/> 
    </proto> 
    <proto name="s1ap" showname="S1 Application Protocol" size="45" pos="78"> 
    <field name="per.choice_index" showname="Choice Index: 0" hide="yes" size="1" pos="78" show="0" value="00"/> 
    <field name="s1ap.S1AP_PDU" showname="S1AP-PDU: initiatingMessage (0)" size="45" pos="78" show="0" value="000b402900000300000005c007c03ae900080003403b53001a0012113743f99f9500075d010605f070c04070c1"> 
     <field name="s1ap.initiatingMessage_element" showname="initiatingMessage" size="45" pos="78" show="" value=""> 
     <field name="s1ap.procedureCode" showname="procedureCode: id-uplinkTransport (13)" size="1" pos="79" show="13" value="0b"/> 
     </field> 
    </field> 
    </proto> 
</packet> 
<!-- more <packet>s here --> 

Answer 1

DOM이 바람직합니다. 특히 문제가 주어진다면 단 한 번의 통과 만 있으면됩니다.

require 'nokogiri' 

class PacketFilter < Nokogiri::XML::SAX::Document 
    def initialize 
    reset 
    end 

    def end_document 
    puts 'the document has ended' 
    end 

    def start_element(name, attributes = []) 
    case name 
    when 'packet' 
     @in_packet = true 
    when 'proto' 
     @have_s1ap = @in_packet && attribute_value(attributes, 'name') == 's1ap' 
    when 'field' 
     case attribute_value(attributes, 'name') 
     when 's1ap.procedureCode' 
     @procedure_code = attribute_value(attributes, 'showname') 
     when 'frame.time_epoch' 
     @epoch_time = attribute_value(attributes, 'showname') 
     end 
    end 
    end 

    def end_element(name) 
    if name == 'packet' 
     puts "#{@procedure_code}, #{@epoch_time}" if @have_s1ap 
     reset 
    end 
    end 

    private 

    def attribute_value(attributes, name) 
    attributes.reduce(nil) do |value, assoc| 
     assoc[0] == name ? assoc[1] : value 
    end 
    end 

    def reset 
    @in_packet = false 
    @have_s1ap = false 
    @procedure_code = nil 
    @epoch_time = nil 
    end 
end 

parser = Nokogiri::XML::SAX::Parser.new(PacketFilter.new) 
parser.parse($stdin) 

당신이 data.xml에 데이터 샘플을 붙여 넣을 경우

및 위의 루비 slap.rb로 : 대신 루프의

$ cat data.xml | ruby poke.rb 
procedureCode: id-downlinkNASTransport (11), Epoch Time: 1474267259.184197000 seconds 
the document has ended 

Answer 2

여기

는 SAX로 XML을 스트리밍하여 작업을 수행 코드 모든 노드를 통해 packet 요소 만 반복하고 기준에 맞지 않는 요소는 건너 뛸 수 있습니다. 이 경우 요소의 모두 대신에 packet 개의 요소 만 수행되므로 속도가 훨씬 빨라야합니다. 일부가 건너 뛰는 추가 조치를 처리 할

data = [] 
file = 'some_file.xml' 
doc = Nokogiri::XML.fragment(File.read(file)) # use `read` instead of `open` 
doc.xpath('packet').each do |packet| 
    next if !packet.css("field[name='s1ap.procedureCode']") ## do nothing if the <packet> is not of s1ap type 
    epochTime = packet.css("field[name='frame.time_epoch']").first["show"].to_i 
    procedureCode = procedureCode_node = packet.css("field[name='s1ap.procedureCode']").first["show"].to_i 
    data << { epochTime: epochTime, procedureCode: procedureCode } 
end 

» data 
=> [{:epochTime=>1474267259, :procedureCode=>11}