Verena

...

KI-SEMINAR

Projektdokumentation

Kurzbeschreibung (EN)

THE OFFICE (working title), 2019
Humans and plants live on different timescales. This is certainly one of the reasons why in everyday life, plants might often seem static and object-like to us. »The Office« (2019) makes use of existing video footage that covers large time spans, in this case, popular long-term tv shows. These productions showcase office and apartment interiors over a longer period of time, often also including houseplants which are mainly used for decorative purposes. Some of these series run over years or even decades and thus comprise many hours of footage. Convolutional neural networks are used as a tool to detect scenes involving houseplants while processing large quantities of the given video material. Selected scenes are compiled into a time-lapse movie which documents plant growth over a long period of time. While the lively movement and growth of the normally passively seeming plants becomes visible, the human activities become blurry and fade into the background.
Keywords: plants, plant-human-relationships, time, timescales, time-lapse, video, neural networks

Hintergrund/Research

Projekte Pflanzen & KI

• Botanica Variegata (2019?) von Sjoerd Ter Borg

Projekte Environment & KI

• Biophillic Vision - Experiment 1: experimenting with machine learning to remove cars from video footage von Chris Harris

Technische Umsetzung/Vorgehensweise

Erste Schritte

• Convolutional Neural Networks for object detection (in videos)
• Heartbeat Tutorial Part 1: Detecting objects in videos and camera feeds using Keras, OpenCV, and ImageAI
  • using a Python library called ImageAi
  • using a pretrained YOLOv3 computer vision model that can recognize 80 different objects, including "potted plants"
  • TO DO: installing a number of python libraries and ImageAi
• Heartbeat Tutorial Part 2: Analyze and Visualize Detected Video Objects Using Keras and ImageAI

Aktueller Stand

Working with the pretrained network

• analyze video frame by frame (works for example with .mp4 and .m4v)
• detect custom objects, in this case of the category "potted plant"
• output a list called "detected frames" that for each frame in the video contains either a 0 or 1 to indicate whether a plant has or has not been detected in that frame. For example (for a video with 24 frames):

[0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0]

• transform this list into a numpy array
• transform this array into another array called "changeArray" only indicates the positions where something changes in the array (if values jump from 0 to 1, or from 1 to 0 or in other words detect the beginning and end of the plant sequence
• reshape array into an array called "changeArrayReshaped" with 2 columns with each row containing only the start and stop frame of the plant sequences

[[ 724 736]

[1716 1717]
[1734 1739]
[1742 1807]
[1809 1812]
[2073 2075]
[2077 2102]
[3260 3309]
[3344 3376]
[3416 3424]
[3497 3526]]

• write this data into a csv file

Preparation of training, test and validation data

• take last DVD from each season (the selected tv series comprises 9 seasons in total) and reserve it for generating the training, test and validation data (this material will not be used for the final analysis)
• format of original material: .mp4; H.264; 720x406; 44100 Hz; 25fps; ...
• convert all .mp4 files from last DVD to 15fps to reduce file size in regard to analysis (we probably do not need so many frames per second as images will be very similar)
• analyze video files (see heartbeat tutorial) and output detected frames by custom objects detection ('potted plant') in the form of a csv file
• use detected frames data in csv file to cut .mp4 files into multiple shorter files (sequences containing plants)
• handsort all video sequences (.mp4 files) into subfolders, e.g.:

[ all_HANDSORTED_original ]
   [ plant_center ]
      [ TO_S2_15fps.mp4_sub_5.mp4 ]
      [ TO_S2_15fps.mp4_sub_6.mp4 ]
      [ TO_S2_15fps.mp4_sub_20.mp4 ]
      ...
   [ plant_chefofficefront ]
   [ plant_receptiontop ]
   ...

• rename video files in subfolders with consecutive file names for batch processing

[ renamed_plant_center ]
   [ plant_center_1.mp4 ]
   [ plant_center_2.mp4 ]
   [ plant_center_3.mp4 ]
   ...
   [ plant_center_268.mp4 ]
[ renamed_plant_chefofficefront ]
[ renamed_plant_receptiontop ]
...

• batch processing: convert all video files in plant-specific subfolders to images(.jpg)
• rename again into images with consecutive numbers

[ plant_center_images_toCrop_renamed ]
   [ plant_center_1.jpg ] 
   [ plant_center_2.jpg ] 
   [ plant_center_3.jpg ] 
   ...
   [ plant_center_2186.jpg ]

• analyze single images (jpgs) with code based on (adapted) imageAi tutorial "Object Detection with 10 lines of code.ipynb"
• per batch processed image: output box points info of detected plant and, within each loop, crop image according to box points and save in newly created folder

Cropped images final output:

• plant_center (ca. 2200 images)
• plant_chefofficefront(ca. 1300 images)
• plant_conferencefront (ca. 600 images)
• plant_conferenceinside (ca. 300 images)
• plant_receptiontop (ca. 738 images)

Nächste Schritte

• train custom model based on collected images
• collecting and processing larger quantities of suitable video material
• further automatizing the process of preselecting plant scenes, splitting the

detected plants into different classes, and making a new cut based on these classes in order to generate visually coherent video material

• final selection of interesting output movies for presentation purposes
• video post production (if neccessary)

Links zum Thema KI

• Don’t let industry write the rules for AI
• The AI Text Generator That's Too Dangerous to Make Public

Evolutionary Algorithms

• Evolutionary algorithm outperforms deep-learning machines at video games, MIT Technology Review, 06/2018

Other Research

Teaching & Non-Teaching

• Meet the school with no classes, no classrooms and no curriculum, Medium, 05/2019