nvidia nx平台 Gstreamer tcpserverink延迟2-3秒问题调试1

1. 前言
2.调整nvv4l2h265enc属性
3. 尝试UDP传输
4.  更换服务器命令

1. 前言

试图发送一个视频流与tcp
但得到2-3秒的延迟。
接收到的流有时在一个灰色图像上停止，然后接收到加速帧的突发。
在Jetpack 4.5.1上有一个Jetson NX Xavier devkit。
我将它与nvpmodel - m2和jetson_clocks -fan一起使用。
使VIC在这里描述的最大时钟运行

v4l2-ctl --list-formats-ext -d /dev/video0

ioctl: VIDIOC_ENUM_FMT
Index : 0
Type : Video Capture
Pixel Format: ‘YUYV’
Name : YUYV 4:2:2
Size: Discrete 1920x1080
Interval: Discrete 0.017s (60.000 fps)

pipeline :

std::string gst_out = "appsrc ! videoconvert ! queue ! nvvidconv ! 
nvv4l2h265enc maxperf-enable=1 ! h265parse 
! matroskamux ! tcpserversink port=AAAA 
host=XXX.XXX.XXX.XXX sync=false async=false";

codec：

cv::VideoWriter out(gst_out, cv::VideoWriter::fourcc('F', 'M', 'P', '4'), 
double(fps), cv::Size(int(w), int(h)));

尝试用video/x-raw删除视频转换，
格式=YUYV，宽度=1920，
高度=1080，帧率=60/1，
但它不工作与这个设置。

用fpsdisplaysink检查视频流

gst-launch-1.0 v4l2src ! videoconvert ! fpsdisplaysink text-overlay=0 video-sink=fakesink sink=0 -v
Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
Setting pipeline to PLAYING …
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)60/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)60/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)60/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)60/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)60/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)60/1, format=(string)YUY2, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)bt709, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 32, dropped: 0, current: 61,95, average: 61,95
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 63, dropped: 0, current: 60,01, average: 60,98
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 93, dropped: 0, current: 60,00, average: 60,66
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 124, dropped: 0, current: 60,00, average: 60,49
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 154, dropped: 0, current: 60,00, average: 60,40
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 185, dropped: 0, current: 60,00, average: 60,33
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 216, dropped: 0, current: 60,01, average: 60,28
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 247, dropped: 0, current: 59,99, average: 60,25
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 278, dropped: 0, current: 60,00, average: 60,22
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 309, dropped: 0, current: 60,00, average: 60,20
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 339, dropped: 0, current: 60,00, average: 60,18
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 370, dropped: 0, current: 60,00, average: 60,16
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 401, dropped: 0, current: 60,00, average: 60,15
^Chandling interrupt.
Interrupt: Stopping pipeline …
Execution ended after 0:00:07.000306499
Setting pipeline to PAUSED …
Setting pipeline to READY …
Setting pipeline to NULL …
Freeing pipeline …

2.调整nvv4l2h265enc属性

调整nvv4l2h265enc中的两个属性并尝试:

iframeinterval      : Encoding Intra Frame occurance frequency
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 30
  idrinterval         : Encoding IDR Frame occurance frequency
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 256

默认设置可能不适合流
可以尝试idrinterval=30或15

尝试为idrinterval(15, 30)和
iframeinterval(5, 10, 20, 50, 100, 200, 300, 600)
设置多个值，但没有任何变化

以下是我的代码:

// Convert to string
#define SSTR( x ) static_cast< std::ostringstream & >( \
( std::ostringstream() << std::dec << x ) ).str()

int main(int argc, char** argv)
{
    cv::VideoCapture invid;
    if (!invid.open(0))
    {
        std::cout << "error opening video file" << std::endl;
        return -1;
    }

    std::cout << "camera open " << std::endl;

    cv::Mat src;
    invid >> src;

    int w, h, fps, format = 0;
    w = invid.get(cv::CAP_PROP_FRAME_WIDTH);    //1920 
    h = invid.get(cv::CAP_PROP_FRAME_HEIGHT);   //1080 
    fps = invid.get(cv::CAP_PROP_FPS);          //60 
    format = invid.get(cv::CAP_PROP_FORMAT);    //0    //doesn't seems to get the right format
    /*v4l2-ctl --list-formats-ext -d /dev/video0  // Camera details
    ioctl: VIDIOC_ENUM_FMT
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'YUYV'
        Name        : YUYV 4:2:2
            Size: Discrete 1920x1080
                Interval: Discrete 0.017s (60.000 fps)*/

    std::cout << "frame width : " << w << " frame height : " << h << " fps : " << fps << " format : " << format << std::endl;

    std::string gst_out = "appsrc ! videoconvert ! queue ! nvvidconv ! nvv4l2h265enc maxperf-enable=1 idrinterval=30 iframeinterval=5 ! h265parse ! matroskamux ! tcpserversink port=8554 host=192.168.1.29 sync=false async=false"; //marche --> attention 脿 l'addresse ip
    cv::VideoWriter out(gst_out, cv::VideoWriter::fourcc('F', 'M', 'P', '4'), double(fps), cv::Size(int(w), int(h)));

    using std::chrono::high_resolution_clock;                           // Calcul loop time
    using std::chrono::duration_cast;
    using std::chrono::duration;
    using std::chrono::milliseconds;
    duration<double, std::milli> time_ms;

    cv::Mat img;

    if(!out.isOpened())
    {
        std::cout << "Failed to open output" << std::endl;
        return 0;
    }

    if(invid.isOpened())
    {
        auto t1 = high_resolution_clock::now();
        //Process
        for(;;)
        {
            auto t2 = high_resolution_clock::now();
            time_ms = t2 - t1;
            //std::cout << "time_ms : " << time_ms.count() << " ms." << std::endl;  
            t1 = t2; 

            float fps_calculated = 1000/time_ms.count(); // Calculate Frames per second (FPS)

            invid >> img;
            /*if (!invid.read(img))
                return 0;*/
            if(img.empty() != 0)
                return 0;

            // Display FPS on frame
            putText(img, "FPS : " + SSTR(int(fps_calculated)), cv::Point(100,50), cv::FONT_HERSHEY_SIMPLEX, 0.75, cv::Scalar(50,170,50), 2);
            std::cout << "fps_calculated : " << fps_calculated << std::endl;
            std::cout << "time_ms.count() : " << time_ms.count() << std::endl;

            //cv::imshow("azert", img);                             

            out.write(img);
            if(cv::waitKey(10) == 27)
                break; // stop capturing by pressing ESC
        }
    }
}

3. 尝试UDP传输

尝试了UDP在TX2 NX作为服务器，
x86 PC作为客户端。
服务器命令:

gst-launch-1.0 videotestsrc is-live=1 
! video/x-raw,width=1280,height=720 
! timeoverlay valignment=4 halignment=1 
! nvvidconv ! 'video/x-raw(memory:NVMM),width=1280,height=720' 
! tee name=t ! nvv4l2h264enc insert-sps-pps=1 idrinterval=15 
! h264parse ! rtph264pay 
! udpsink host=10.19.106.10 port=5000 sync=0 t. 
! queue ! nvegltransform ! nveglglessink sync=0

客户端命令:

gst-launch-1.0 udpsrc port=5000 
! 'application/x-rtp,encoding-name=H264,payload=96' 
! rtph264depay ! avdec_h264 ! xvimagesink sync=0

结果如下
(左边是TX2 NX，右边是x86 PC):

4. 更换服务器命令

尝试以下情况下的服务器命令:

gst-launch-1.0 videotestsrc in UDP
gst-launch-1.0 videotestsrc in TCP
gst-launch-1.0 OpenCV in UDP

看看能不能通过这些弄清楚延迟的原因
运行gst-inspect-1.0 nvv4l2h265enc获取所有属性

但我的Ubuntu 18.04 x86 PC没有得到视频测试:

gst-launch-1.0 udpsrc port=5000 ! 'application/x-rtp,encoding-name=H264,payload=96' ! rtph264depay ! avdec_h264 ! xvimagesink sync=0
Définition du pipeline à PAUSED...
Le pipeline est actif et n’a pas besoin de phase PREROLL…
Passage du pipeline à la phase PLAYING…
New clock: GstSystemClock
^Chandling interrupt.
Interruption : arrêt du pipeline…
Execution ended after 0:00:17.859331593
Définition du pipeline à PAUSED...
Définition du pipeline à READY (prêt)…
Définition du pipeline à NULL…
Libération du pipeline…

没有图像显示。

还有gst-launch-1.0 nvv4l2h265enc在我的NX Xavier:

$ gst-launch-1.0 nvv4l2h265enc
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE
Opening in BLOCKING MODE 
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:07.792176179
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

命令不正确
应该是gst-inspect-1.0