VID2XML

Extracting XML Data From Mobile Screencasts

vid2XML

vid2XML is a research tool that automatically extract a complete XML data from mobile programming videos. It relies on the visual and textual features of video frames and utilizes an OCR engine.

vid2XML Architecture

vid2XML Architecture

More Details

@inproceedings{alahmadi2022vid2xml,
  Author = {Alahmadi, Mohammad}
  Title = {Automatic Extraction of a Complete XML Data from Mobile Programming Screencasts},
  Booktitle = {IEEE Transactions on Software Engineering},
  Pages = {},
  Publisher = {IEEE},
  Year = {2022}
}

The Entire Dataset with Source Code

We provide all of our data, models, source code, and results. You can download the zip here:

VID2XML Dataset Updated on May 2022

The vid2XML's source code Dependancies

To run vid2XML and re-produce the results, you need to first install some depandancies and run main.py file:

We recommend to use conda. and create a new environment before you install the following packages

fuzzywuzzy==0.18.0
numpy==1.20.1
requests==2.25.1
tensorflow==2.7.0
fuzzywuzzy==0.18.0
cv2==4.5.3

You would also need to obtain a key from Google Vision for the OCR to run. Use the following link for instructions about setting up google Vision Quick Start

activity_main.xml 

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
	xmlns:app="http://schemas.android.com/apk/res-auto"
	xmlns:tools="http://schemas.android.com/tools"
	android:layout_width="match_parent"
	android:layout_height="match_parent"
	android:orientation="vertical"
	android:padding="16dp"
	tools:context="com.codinginflow.firestoreexampleproject.MainActivity">

	<EditText
		android:id="@+id/edit_text_title"
		android:layout_width="match_parent"
		android:layout_height="wrap_content"
		android:hint="Title"
		android:inputtype="text" />

	<EditText
		android:id="@+id/edit_text_description"
		android:layout_width="match_parent"
		android:layout_height="wrap_content"
		android:hint="Description"
		android:inputtype="text" />

	<Button
		android:layout_width="match_parent"
		android:layout_height="wrap_content"
		android:onclick="saveNote"
		android:text="Save" />

	<Button
		android:layout_width="match_parent"
		android:layout_height="wrap_content"
		android:onclick="loadNote"
		android:text="Load" />

	<Button
		android:layout_width="match_parent"
		android:layout_height="wrap_content"
		android:onclick="updateDescription"
		android:text="update description" />

	<Button
		android:layout_width="match_parent"
		android:layout_height="wrap_content"
		android:text="Delete description"
		android:onclick="deleteDescription"/>

	<Button
		android:layout_width="match_parent"
		android:layout_height="wrap_content"
		android:text="Delete note"
		android:onclick="deleteNote" />

	<TextView
		android:id="@+id/text_view_data"
		android:layout_width="wrap_content"
		android:layout_height="wrap_content"
		android:textSize="20sp" />
</LinearLayout>

activity_main.xml 

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
	xmlns:app="http://schemas.android.com/apk/res-auto"
	xmlns:tools="http://schemas.android.com/tools"
	android:layout_width="match_parent"
	android:layout_height="match_parent"
	tools:context="com.codinginflow.viewflipperexample.MainActivity">

	<ViewFlipper
		android:id="@+id/view_flipper"
		android:layout_width="match_parent"
		android:layout_height="match_parent"
		android:inAnimation="@android:anim/slide_in_left"
		android:outAnimation="@android:anim/slide_out_right">

		<Button
			android:layout_width="wrap_content"
			android:layout_height="wrap_content"
			android:text="Button"
			android:layout_gravity="center"/>

		<ImageView
			android:layout_width="wrap_content"
			android:layout_height="wrap_content"
			android:src="@mipmap/ic_launcher"
			android:layout_gravity="center"/>

		<LinearLayout
			android:layout_width="wrap_content"
			android:layout_height="wrap_content"
			android:orientation="vertical"
			android:layout_gravity="center">

			<TextView
				android:layout_width="wrap_content"
				android:layout_height="wrap_content"
				android:text="TextView1"
				android:textSize="20sp"/>

slide_in_right.xml 

<?xml version="1.0" encoding="utf-8"?>
<set xmlns:android="http://schemas.android.com/apk/res/android">

	<translate
		android:duration="@android:integer/config_mediumAnimTime"
		android:fromXDelta="50%"
		android:toXDelta="" />

	<alpha
		android:duration="@android:integer/config_mediumAnimTime"
		android:fromAlpha="0.0"
		android:toAlpha="1.0" />
</set>

slide_out_left.xml 

<?xml version="1.0" encoding="utf-8"?>
<set xmlns:android="http://schemas.android.com/apk/res/android">

	<translate
		android:duration="@android:integer/config_mediumAnimTime"
		android:fromXDelta="0"
		android:toxDelta="-50%p" />

	<alpha
		android:duration="@android:integer/config_mediumAnimTime"
		android:fromAlpha="1.0"
		android:toAlpha="0.0" />
</set>

activity_main.xml 

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
	xmlns:app="http://schemas.android.com/apk/res-auto"
	xmlns:tools="http://schemas.android.com/tools"
	android:layout_width="match_parent"
	android:layout_height="match_parent"
	tools:context="com.codinginflow.constraintlayoutexample.MainActivity">

	<TextView
		android:layout_width="wrap_content"
		android:layout_height="wrap_content"
		android:text="Hello world!"
		app:layout_constraintBottom_toBottomof="parent"
		app:layout_constraintLeft_toLeftof="parent"
		app:layout_constraintRight_toRightof="parent"
		app:layout_constraintTop_totopof="parent" />
</RelativeLayout>

activity_launch.xml 

<?xml version="1.0" encoding="utf-8"?>
<android.support.constraint.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
	xmlns:app="http://schemas.android.com/apk/res-auto"
	xmlns:tools="http://schemas.android.com/tools"
	android:layout_width="match_parent"
	android:layout_height="match_parent"
	tools:context=".Launch">

	<TextView
		android:layout_width="match_parent"
		android:layout_height="match_parent"
		android:text="Welcome To DN Tutorial"
		android:textAlignment="center"
		android:layout_marginTop="25sp"
		android:textSize="30sp"/>
</android.support.constraint.ConstraintLayout>

activity_main.xml 

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
	xmlns:app="http://schemas.android.com/apk/res-auto"
	xmlns:tools="http://schemas.android.com/tools"
	android:layout_width="match_parent"
	android:layout_height="match_parent"
	android:background="@color/colorPrimary"
	tools:context=".MainActivity"
	android:layout_gravity="center"
	android:gravity="center"
	android:orientation="vertical">

	<ImageView
		android:layout_width="wrap_content"
		android:layout_height="wrap_content"
		android:src="@drawable/logo"/>

	<TextView
		android:layout_width="match_parent"
		android:layout_height="wrap_content"
		android:text="@string/dn_tutorial"
		android:textAlignment="center"
		android:layout_marginTop="20sp"
		android:textColor="#FFF"
		android:textStyle="bold"
		android:textSize="25sp"/>

	<ProgressBar
		android:layout_width="wrap_content"
		android:layout_height="wrap_content"
		android:layout_marginTop="20sp"/>
</LinearLayout>

activity_2.xml 

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/re: 2
	xmlns:app="http://schemas.android.com/apk/res-auto"
	xmlns:tools="http://schemas.android.com/tools"
	android:layout_width="match_parent"
	android:layout_height="match_parent"
	tools:context="com.example.application.myapplication.MainAct:

	<TextView
		android:layout_width="wrap_content"
		android:layout_height="wrap_content"
		android:text="Activity 2"
		android:layout_centervertical="true"
		android:layout_centerHorizontal="true" />
</RelativeLayout>
</android.support.constraint.ConstraintLayout>

activity_main.xml 

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/re::
	xmlns:app="http://schemas.android.com/apk/res-auto"
	xmlns:tools="http://schemas.android.com/tools"
	android:layout_width="match_parent"
	android:layout_height="match_parent"
	tools:context="com.example.application.myapplication.MainAct:

	<TextView
		android:layout_width="wrap_content"
		android:layout_height="wrap_content"
		android:text="Activity 1"
		android:layout_centervertical="true"
		android:layout_centerHorizontal="true"
		android:id="@+id/textView" />

	<Button
		android:id="@+id/button"
		android:layout_width="wrap_content"
		android:layout_height="wrap_content"
		android:text="open activity 2"
		android:layout_below="@+id/textView"
		android:layout_centerHorizontal="true" />
</RelativeLayout>

activity_main.xml 

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
	xmlns:app="http://schemas.android.com/apk/res-auto"
	xmlns:tools="http://schemas.android.com/tools"
	android:layout_width="match_parent"
	android:layout_height="match_parent"
	tools:context="com.example.application.example.MainActivity">

	<Button
		android:id="@+id/button"
		android:layout_width="wrap_content"
		android:layout_height="wrap_content"
		android:text="open dialog"
		android:layout_centerVertical="true"
		android:layout_centerHorizontal="true" />
</RelativeLayout>

<Button
	android:layout_width="wrap_content"
	android:layout_height="wrap_content"
	android:text="opl" />
</RelativeLayout>

activity_main.xml 

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout
	android:orientation="vertical"
	xmlns:android="http:77schemas.android.com/apk/res/anarola"
	xmlns:app="http://schemas.android.com/apk/res-auto"
	xmlns:tools="http://schemas.android.com/tools"
	android:layout_width="match_parent"
	android:layout_neignt="match_parent"
	tools:context=".MainActivity">

	<TextView
		android:id="@+id/time TV"
		android:layout_width="match_parent"
		android:layout_height="wrap_content"
		android:text="UU:UU:UU"
		android:padding="10dp"
		android:textColor="@color/black"
		android:textSize="60sp"
		android:textAlignment="center"/>

	<androidx.constraintlayout.widget.ConstraintLayout
		android:layout_width="match_parent"
		android:layout_height="wrap_content"
		android:padding="10dp">

		<com.google.android.material.button.MaterialButton
			android:id="@+id/startStopButton"
			android:layout_width="wrap_content"
			android:layout_height="wrap_content"
			android:text="start"
			app:layout_constraintStart_toStartOf="parent"
			app:layout_constraintTop_toTopOf="parent" />

		<com.google.android.material.button.MaterialButton
			android:id="@+id/resetButton"
			android:layout_width="wrap_content"
			android:layout_height="wrap_content"
			android:text="reset"
			app:icon="@drawable/ic_baseline_replay_24"
			app:layout_constraintEnd_toEndOf="parent"
			app:layout_constraintTop_toTopOf="parent" />
	</androidx.constraintlayout.widget.ConstraintLayout>
	earLayout>
	yout androidx.constraintlayout.widget.ConstraintLayout com.google.android.material.button.MaterialButton />
</LinearLayout>

AndroidManifest.xml 

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
	package="code.with.cal.timeronservicetutorial">

	<application
		android:allowBackup="true"
		android:icon="@mipmap/ic_launcher"
		android:label="TimerOnService Tutorial"
		android:roundIcon="@mipmap/ic_launcher_round"
		android:supportsRtl="true"
		android:theme="@style/Theme.TimerOnServiceTutorial">

		<activity android:name=".MainActivity">

			<intent-filter>

				<action android:name="android.intent.action.MAIN" />

				<category android:name="android.intent.category.LAUNCHER" />
			</intent-filter>
		</activity>

		<service android:name=".TimerService"/>
	</application>
</manifest>

ic_baseline_pause_24.xml 

<vector xmlns:android="http://schemas.android.com/apk/res/android"
	android width="24dp"
	android:height="24dp"
	android:viewportWidth="24"
	android:viewportHeight="24"
	android:tint="?attr/colorControlNormal">

	<path
		android:fillcolor="@android:color/white"
		android:pathData="M6, 19h4L10,5L6,5v14zM14,5v14h4L18,5h-42"/>
</vector>

main.axml 

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
	android:orientation="vertical"
	android:layout_width="fill_parent"
	android:layout_height="fill_parent"
	android:minwidth="25px"
	android:inheight="25px">

	<ListView
		android:layout_width="match_parent"
		android:layout height="match_parent"
		android:id="@+id/listView1" />
</LinearLayout>

activity_main.xml 

<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
	xmlns:app="http://schemas.android.com/apk/res-auto"
	xmlns:tools="http://schemas.android.com/tools"
	android:layout_width="match_parent"
	android:layout_height="match_parent"
	android:padding="16dp"
	tools:context=".MainActivity">

	<Spinner
		android:id="@+id/spMonths"
		android:layout_width="wrap_content"
		android:layout_height="wrap_content"
		android:entries="@array/months"
		app:layout_constraintBottom_toBottomóf="parent"
		app:layout_constraintEnd_toEndof="parent"
		app:layout_constraintHorizontal_bias="0.5"
		app:layout_constraintStart_toStartof="parent"
		app:layout_constraintTop_toTopOf="parent" />
</androidx.constraintlayout.widget.ConstraintLayout>

strings.xml 

<resources>

	<string name="app_name">Tutorials</string>

	<string-array name="months">

		<item>January</item>

		<item> February</item>

		<item>March</item>

		<item>April</item>

		<item>May</item>

		<item> June</item>

		<item>July</item>

		<item>August</item>

		<item>September</item>

		<item>October</item>

		<item>November</item>

		<item>December</item>
	</string-array>
</resources>

main.axml 

<?xml version="1.0" encoding="utf-8"?>
<android.support: 17.widget:GridLayout xmlns:android="http://schemas.android.com/apk/res/android"
	xmlns:app="http://schemas.android.com/apk/res-auto"
	android:layout_width="match_parent"
	android:layout_height="match_parent"
	app:columnCount="2"
	app:rowCount="3">

	<TextView
		app:layout_columnWeight="1"
		app:layout_rowWeight="1"
		app:layout_columnSpan="2"
		android:background="#F44336"
		android:textSize="70sp"
		android:text="1st" />

	<TextView
		app:layout_columnWeight="1"
		app:layout_rowWeight="1"
		android:background="#2196F3"
		android:textSize="70sp"
		android:text="2nd" />

	<TextView
		app:layout_columnWeight="1"
		app:layout_rowWeight="1"
		app:layout_rowSpan="2"
		android:background="#4CAF50"
		android:textSize="70sp"
		android:text="3rd" />

	<TextView
		android:layout_columnWeight="1"
		android:layout_rowWeight="1"
		android:background="#FFC107"
		android:textSize="70sp"
		android:text="4th" />
</android.support.v7.widget.GridLayout>

Examples of Merging Two XML Data

Input1 Input2 Output 
<Button
  android:id="@+id/nextButton"
  android:text="Next"

  android:text="Next"
  android:onclick="nextView" />
</LinearLayout>

<Button
  android:id="@+id/nextButton"
  android:text="Next"
  android:onclick="nextView" />
</LinearLayout>

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout
  android:layout_width="match_parent"
  android:layout_height="match_parent"
  android:paddingLeft="16dp"
  android:paddingRight="16dp" >
 </LinearLayout>
            
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout
  android:layout_width="match_parent"
  android:layout_height="match_parent"
  android:paddingLeft="16dp"
  android:paddingRight="16dp"
  android:orientation="vertical" >
</LinearLayout>

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout
  android:layout_width="match_parent"
  android:layout_height="match_parent"
  android:paddingLeft="16dp"
  android:paddingRight="16dp"
  android:orientation="vertical" >
</LinearLayout>


<?xml version="1.0" encoding="utf-8"?>
<LinearLayout
  android:orientation="vertical"
  android:layout_width="fill_parent"
  android:layout_height="fill parent"
  android:minWidth="25px"
  android:minHeight="25px">
  <ListView
    android:minwidth="25px"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:id="@+id/listView1" />
</LinearLayout>
            
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout
  android:orientation="vertical"
  android:layout_width="fill_parent"
  android:layout_height="fill parent"
  android:minWidth="100px"
  android:minHeight="25px">
  <ListView
    android:minwidth="25px"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:id="@+id/listView1" />
</LinearLayout>

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout
  android:orientation="vertical"
  android:layout_width="fill_parent"
  android:layout_height="fill parent"
  android:minWidth="100px"
  android:minHeight="25px">
  <ListView
    android:minwidth="25px"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:id="@+id/listView1" />
</LinearLayout>

Examples of Locating XML Region, If Exists

Input Image Line Detection XML Region Localization
vid2XML Architecture vid2XML Architecture vid2XML Architecture
vid2XML Architecture vid2XML Architecture vid2XML Architecture
vid2XML Architecture vid2XML Architecture
Result: non-XML region detected
Reason: None of the detected lines contain XML
Action: Ignore the entire frame

Try It Out (Merging Two XML Data)

Feel free to try out different XML data by changing the two input Texts with any valid XML data.